[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:41 dist-test-slave-t1mc krb5kdc[1900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:41 dist-test-slave-t1mc krb5kdc[1900](info): set up 2 sockets
Sep 26 07:01:41 dist-test-slave-t1mc krb5kdc[1900](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 07:01:42.945770 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:43.064472 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34361
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:43.143528 1913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:43.143766 1913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:43.143805 1913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:43.144945 1913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:43.145011 1913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:43.145035 1913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:43.145064 1913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:43.145092 1913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:43.146533 1913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:43.146823 1913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:43.147038 1913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:43.151804 1919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.151805 1921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.151830 1918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:43.151919 1913 server_base.cc:1034] running on GCE node
I20260926 07:01:43.152282 1913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:43.152577 1913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:43.153726 1913 hybrid_clock.cc:648] HybridClock initialized: now 1790406103153694 us; error 49 us; skew 500 ppm
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:43.156059 1913 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:43.156538 1913 webserver.cc:460] Webserver started at http://127.1.217.126:36235/ using document root <none> and password file <none>
I20260926 07:01:43.156759 1913 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:43.156879 1913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:43.157037 1913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:43.158082 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "04d5508568344dd5b95cd7a711cf7cab"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "b542707b349808156b4bfe7cb2a359ee"
server_key_iv: "af5349770b8bff42127c0bc36a8edee3"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.158468 1913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "04d5508568344dd5b95cd7a711cf7cab"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "b542707b349808156b4bfe7cb2a359ee"
server_key_iv: "af5349770b8bff42127c0bc36a8edee3"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.159811 1913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:43.160533 1928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.160702 1913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:43.160825 1913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "04d5508568344dd5b95cd7a711cf7cab"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "b542707b349808156b4bfe7cb2a359ee"
server_key_iv: "af5349770b8bff42127c0bc36a8edee3"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.160912 1913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:43.168299 1913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:43.170091 1913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:43.170258 1913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:43.174166 1913 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34361
I20260926 07:01:43.174196 1980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34361 every 8 connection(s)
I20260926 07:01:43.174528 1913 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:43.175315 1981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:43.177498 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab: Bootstrap starting.
I20260926 07:01:43.178018 1981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:43.178247 1981 log.cc:824] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:43.178797 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 1913
I20260926 07:01:43.178903 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:43.178952 1981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab: No bootstrap required, opened a new log
I20260926 07:01:43.179661 1893 external_mini_cluster.cc:1428] Setting key 9f685a511eb2223f4161d456988973c4
I20260926 07:01:43.181248 1981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } }
I20260926 07:01:43.181424 1981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:43.181461 1981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d5508568344dd5b95cd7a711cf7cab, State: Initialized, Role: FOLLOWER
I20260926 07:01:43.181586 1981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } }
I20260926 07:01:43.181665 1981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:43.181696 1981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:43.181742 1981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:43.182904 1981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } }
I20260926 07:01:43.183032 1981 leader_election.cc:302] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d5508568344dd5b95cd7a711cf7cab; no voters:
I20260926 07:01:43.183200 1981 leader_election.cc:288] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:43.183233 1986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:43.183411 1986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [term 1 LEADER]: Becoming Leader. State: Replica: 04d5508568344dd5b95cd7a711cf7cab, State: Running, Role: LEADER
I20260926 07:01:43.183454 1981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:43.183538 1986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } }
I20260926 07:01:43.184005 1987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d5508568344dd5b95cd7a711cf7cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } } }
I20260926 07:01:43.184103 1987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:43.184018 1988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d5508568344dd5b95cd7a711cf7cab. Latest consensus state: current_term: 1 leader_uuid: "04d5508568344dd5b95cd7a711cf7cab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d5508568344dd5b95cd7a711cf7cab" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34361 } } }
I20260926 07:01:43.184264 1988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:43.184414 1995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:43.185209 1995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:43.186784 1995 catalog_manager.cc:1347] Generated new cluster ID: f3b8b6c21f294ca991d5d0466d553f8b
I20260926 07:01:43.186833 1995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:43.197984 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.180544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.1.217.126:34361)
0926 07:01:43.180914 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.180922 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.180967 (+ 45us) server_negotiation.cc:407] Connection header received
0926 07:01:43.183501 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.183508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.183565 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.183722 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.186210 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.187599 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.189066 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.189696 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.189818 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.193995 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.193998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.193999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.194009 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.195939 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.196463 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.196466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.196466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.196567 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.196878 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.196879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.196879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.197157 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.197303 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.197611 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.197740 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":91,"threads_started":1}
I20260926 07:01:43.205916 1995 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:43.206666 1995 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:43.218511 1995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d5508568344dd5b95cd7a711cf7cab: Generated new TSK 0
I20260926 07:01:43.218725 1995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:43.251808 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:43.332885 2007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:43.333091 2007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:43.333127 2007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:43.334244 2007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:43.334309 2007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:43.334362 2007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:43.335777 2007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:43.336028 2007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:43.336256 2007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:43.341292 2013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.341286 2012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.341320 2015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:43.341429 2007 server_base.cc:1034] running on GCE node
I20260926 07:01:43.341614 2007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:43.341813 2007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:43.342945 2007 hybrid_clock.cc:648] HybridClock initialized: now 1790406103342928 us; error 34 us; skew 500 ppm
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:43.345412 2007 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:43.345849 2007 webserver.cc:460] Webserver started at http://127.1.217.65:34493/ using document root <none> and password file <none>
I20260926 07:01:43.346091 2007 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:43.346158 2007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:43.346292 2007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:43.347328 2007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "39a04a4efdbe4a5eb824b301bde93a65"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "ffc6000c1db90f901cc4c8eb455d80ea"
server_key_iv: "2323677e263a32ee8ea593b699cada67"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.347689 2007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "39a04a4efdbe4a5eb824b301bde93a65"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "ffc6000c1db90f901cc4c8eb455d80ea"
server_key_iv: "2323677e263a32ee8ea593b699cada67"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.348999 2007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:01:43.349675 2022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.349865 2007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:43.349954 2007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "39a04a4efdbe4a5eb824b301bde93a65"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "ffc6000c1db90f901cc4c8eb455d80ea"
server_key_iv: "2323677e263a32ee8ea593b699cada67"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.350024 2007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:43.361464 2007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:43.363261 2007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:43.363451 2007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:43.363708 2007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:43.364012 2007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:43.364044 2007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.364066 2007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:43.364110 2007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.370021 2007 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:44537
I20260926 07:01:43.370117 2135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:44537 every 8 connection(s)
I20260926 07:01:43.370433 2007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:43.376859 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2007
I20260926 07:01:43.376962 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:43.377132 1893 external_mini_cluster.cc:1428] Setting key d5ec2a26379325ba36eee2c16f77aac0
I20260926 07:01:43.380051 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.371176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55771 (local address 127.1.217.126:34361)
0926 07:01:43.371238 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.371240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.371449 (+ 209us) server_negotiation.cc:407] Connection header received
0926 07:01:43.371983 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.371984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.372001 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.372051 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.373048 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.373560 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.374554 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.374770 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.374862 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.377131 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.377132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.377133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.377139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.378561 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.379057 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.379058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.379059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.379127 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.379393 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.379394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.379394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.379536 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.379608 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.379854 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.379943 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:01:43.380069 2138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.371159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34361 (local address 127.1.217.65:55771)
0926 07:01:43.371334 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:43.371369 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:43.371867 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:43.372126 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:43.372129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:43.372693 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:43.372953 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.372959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.373664 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.373665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.374465 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.374467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.374871 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.374873 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.374995 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.375522 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:43.375530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:43.377035 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:43.378641 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.378648 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.378650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.378973 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.379208 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.379209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.379210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.379329 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.379603 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:43.379607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:43.379723 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 07:01:43.379805 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:01:43.379916 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 07:01:43.380894 2136 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34361
I20260926 07:01:43.380993 2136 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:43.381186 2136 heartbeater.cc:511] Master 127.1.217.126:34361 requested a full tablet report, sending...
I20260926 07:01:43.381716 1945 ts_manager.cc:194] Registered new tserver with Master: 39a04a4efdbe4a5eb824b301bde93a65 (127.1.217.65:44537)
I20260926 07:01:43.382392 1945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:55771
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:43.401402 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:43.481174 2141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:43.481377 2141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:43.481400 2141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:43.482626 2141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:43.482674 2141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:43.482702 2141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:43.484282 2141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:43.484508 2141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:43.484807 2141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:43.489727 2147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.489732 2146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.489742 2149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:43.490011 2141 server_base.cc:1034] running on GCE node
I20260926 07:01:43.490228 2141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:43.490461 2141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:43.491626 2141 hybrid_clock.cc:648] HybridClock initialized: now 1790406103491602 us; error 36 us; skew 500 ppm
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:43.494200 2141 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:43.494647 2141 webserver.cc:460] Webserver started at http://127.1.217.66:41621/ using document root <none> and password file <none>
I20260926 07:01:43.494834 2141 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:43.494874 2141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:43.494961 2141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:43.495983 2141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "641a357c73e040669f07550616712947"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "35445d8816039ab2533d6cd51127947b"
server_key_iv: "4c2f884b1eecd04f66ebae6d519a9121"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.496285 2141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "641a357c73e040669f07550616712947"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "35445d8816039ab2533d6cd51127947b"
server_key_iv: "4c2f884b1eecd04f66ebae6d519a9121"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.497579 2141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:43.498242 2156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.498431 2141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:43.498513 2141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "641a357c73e040669f07550616712947"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "35445d8816039ab2533d6cd51127947b"
server_key_iv: "4c2f884b1eecd04f66ebae6d519a9121"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.498566 2141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:43.530187 2141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:43.532109 2141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:43.532260 2141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:43.532497 2141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:43.532832 2141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:43.532888 2141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.532932 2141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:43.532961 2141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.538694 2141 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:37117
I20260926 07:01:43.538798 2269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:37117 every 8 connection(s)
I20260926 07:01:43.539089 2141 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:43.546065 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2141
I20260926 07:01:43.546181 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:43.546345 1893 external_mini_cluster.cc:1428] Setting key 1f6e77a23c29b098791746ff3b0dbe51
I20260926 07:01:43.548432 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.539805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:47869 (local address 127.1.217.126:34361)
0926 07:01:43.539894 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.539897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.540071 (+ 174us) server_negotiation.cc:407] Connection header received
0926 07:01:43.540598 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.540599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.540611 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.540671 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.541617 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.542122 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.543113 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.543328 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.543416 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.545508 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.545509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.545510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.545517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.547020 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.547410 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.547411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.547412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.547466 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.547714 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.547715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.547722 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.547835 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.547910 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.548232 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.548298 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:01:43.548641 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.539768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34361 (local address 127.1.217.66:47869)
0926 07:01:43.539945 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:43.539989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:43.540492 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:43.540751 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:43.540754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:43.541331 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:43.541512 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.541517 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.542237 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.542238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.543021 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.543022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.543424 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.543425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.543547 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.544055 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:43.544063 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:43.545411 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:43.547102 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.547109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.547111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.547334 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.547539 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.547540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.547540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.547648 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.547900 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:43.547906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:43.548074 (+ 168us) client_negotiation.cc:769] Sending connection context
0926 07:01:43.548201 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:01:43.548386 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260926 07:01:43.549165 2270 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34361
I20260926 07:01:43.549253 2270 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:43.549438 2270 heartbeater.cc:511] Master 127.1.217.126:34361 requested a full tablet report, sending...
I20260926 07:01:43.549786 1945 ts_manager.cc:194] Registered new tserver with Master: 641a357c73e040669f07550616712947 (127.1.217.66:37117)
I20260926 07:01:43.550240 1945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:47869
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:43.571470 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:43.651068 2275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:43.651262 2275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:43.651285 2275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:43.652403 2275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:43.652442 2275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:43.652469 2275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:43.653972 2275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:43.654204 2275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:43.654439 2275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:43.659440 2280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:43.659546 2275 server_base.cc:1034] running on GCE node
W20260926 07:01:43.659469 2281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:43.659624 2283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:43.659883 2275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:43.660135 2275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:43.661290 2275 hybrid_clock.cc:648] HybridClock initialized: now 1790406103661278 us; error 28 us; skew 500 ppm
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:43.664068 2275 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:43.664594 2275 webserver.cc:460] Webserver started at http://127.1.217.67:42597/ using document root <none> and password file <none>
I20260926 07:01:43.664881 2275 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:43.664961 2275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:43.665096 2275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:43.666070 2275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "41531c2f544348d39de594cd1e52b0fb"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "79e56388fa6c975eed2a880738884fe3"
server_key_iv: "925ae8cc77c37b34af7a3579ff2818ea"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.666375 2275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "41531c2f544348d39de594cd1e52b0fb"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "79e56388fa6c975eed2a880738884fe3"
server_key_iv: "925ae8cc77c37b34af7a3579ff2818ea"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.667632 2275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:43.668260 2290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.668447 2275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:43.668525 2275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "41531c2f544348d39de594cd1e52b0fb"
format_stamp: "Formatted at 2026-09-26 07:01:43 on dist-test-slave-t1mc"
server_key: "79e56388fa6c975eed2a880738884fe3"
server_key_iv: "925ae8cc77c37b34af7a3579ff2818ea"
server_key_version: "encryptionkey@0"
I20260926 07:01:43.668610 2275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:43.686964 2275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:43.688849 2275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:43.689004 2275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:43.689234 2275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:43.689553 2275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:43.689610 2275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.689671 2275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:43.689697 2275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:43.695587 2275 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:35311
I20260926 07:01:43.695641 2403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:35311 every 8 connection(s)
I20260926 07:01:43.696039 2275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:43.704977 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.696740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:47699 (local address 127.1.217.126:34361)
0926 07:01:43.696822 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.696824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.697044 (+ 220us) server_negotiation.cc:407] Connection header received
0926 07:01:43.697555 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.697556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.697568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.697612 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.698525 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.699014 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.700005 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.700195 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.700280 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.702457 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.702459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.702459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.702466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.703609 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.704021 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.704022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.704023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.704083 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.704331 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.704332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.704332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.704444 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.704521 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.704806 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.704884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:01:43.705161 2406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.696712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34361 (local address 127.1.217.67:47699)
0926 07:01:43.696918 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:43.696971 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:43.697455 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:43.697675 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:43.697679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:43.698241 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:43.698426 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.698430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.699113 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.699114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.699908 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:43.699909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.700286 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.700287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:43.700414 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.700953 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:43.700961 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:43.702355 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:43.703706 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.703712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.703714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.703931 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.704149 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:43.704150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:43.704150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:43.704259 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:43.704511 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:43.704518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:43.704654 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 07:01:43.704761 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:01:43.704938 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":118,"threads_started":1}
I20260926 07:01:43.705658 2404 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34361
I20260926 07:01:43.705730 2404 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:43.705922 2404 heartbeater.cc:511] Master 127.1.217.126:34361 requested a full tablet report, sending...
I20260926 07:01:43.706161 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2275
I20260926 07:01:43.706238 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:43.706341 1945 ts_manager.cc:194] Registered new tserver with Master: 41531c2f544348d39de594cd1e52b0fb (127.1.217.67:35311)
I20260926 07:01:43.706394 1893 external_mini_cluster.cc:1428] Setting key 53cf49a2d046bd74c700a22d12a265c9
I20260926 07:01:43.706799 1945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:47699
I20260926 07:01:43.707435 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:43.729300 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.721964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.1.217.65:44537)
0926 07:01:43.722109 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.722114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.722132 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:01:43.722156 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.722158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.722172 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.722245 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.722753 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.723456 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.724137 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.724467 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.724545 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.726726 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.726736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.726736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.726743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.728063 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.728431 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.728434 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.728435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.728493 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.728689 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.728690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.728691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.728884 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.729048 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.729091 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.729168 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260926 07:01:43.729668 2050 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54376
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[1900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:43.750852 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:43.743963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.1.217.65:44537)
0926 07:01:43.744038 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:43.744041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:43.744122 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:01:43.744207 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:43.744208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:43.744217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:43.744291 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:43.744873 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.745360 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.746114 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:43.746339 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:43.746414 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:43.748502 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:43.748503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:43.748504 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:43.748511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:43.749723 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.750100 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.750101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.750102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.750156 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:43.750351 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:43.750352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:43.750352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:43.750452 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:43.750569 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:43.750659 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:01:43.750738 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:01:43.751415 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2007
I20260926 07:01:43.756146 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2141
I20260926 07:01:43.760040 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2275
I20260926 07:01:43.764241 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 1913
2026-09-26T07:01:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1863 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[2418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[2418](info): set up 2 sockets
Sep 26 07:01:43 dist-test-slave-t1mc krb5kdc[2418](info): commencing operation
krb5kdc: starting...
W20260926 07:01:44.804173 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:44 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:44.921645 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46101
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:45.001559 2431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:45.001801 2431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:45.001837 2431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:45.002946 2431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:45.003013 2431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:45.003036 2431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:45.003073 2431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:45.003104 2431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:45.004529 2431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:46101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:45.004813 2431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:45.005035 2431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:45.009636 2437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.009658 2439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.009666 2436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:45.009871 2431 server_base.cc:1034] running on GCE node
I20260926 07:01:45.010090 2431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:45.010314 2431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:45.011483 2431 hybrid_clock.cc:648] HybridClock initialized: now 1790406105011474 us; error 41 us; skew 500 ppm
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:45.013975 2431 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:45.014390 2431 webserver.cc:460] Webserver started at http://127.1.217.126:43833/ using document root <none> and password file <none>
I20260926 07:01:45.014621 2431 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:45.014689 2431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:45.014878 2431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:45.015956 2431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "ce983183358e4c84aec481e67393a3c3"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "97edef807e2782953969d9021153c1c6"
server_key_iv: "042382a1ccc1a3caf9acf3548f8a4689"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.016316 2431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "ce983183358e4c84aec481e67393a3c3"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "97edef807e2782953969d9021153c1c6"
server_key_iv: "042382a1ccc1a3caf9acf3548f8a4689"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.017704 2431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:45.018393 2446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.018560 2431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:45.018659 2431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "ce983183358e4c84aec481e67393a3c3"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "97edef807e2782953969d9021153c1c6"
server_key_iv: "042382a1ccc1a3caf9acf3548f8a4689"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.018734 2431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:45.033345 2431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:45.035568 2431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:45.035730 2431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:45.039574 2431 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:46101
I20260926 07:01:45.039609 2498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:46101 every 8 connection(s)
I20260926 07:01:45.039942 2431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:45.040753 2499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:45.043037 2499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3: Bootstrap starting.
I20260926 07:01:45.043614 2499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:45.043901 2499 log.cc:824] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:45.044589 2499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3: No bootstrap required, opened a new log
I20260926 07:01:45.045974 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2431
I20260926 07:01:45.046067 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:45.046217 1893 external_mini_cluster.cc:1428] Setting key bdc7c5aa540da8bf1343f3283b79ebec
I20260926 07:01:45.046471 2499 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } }
I20260926 07:01:45.046648 2499 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:45.046687 2499 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce983183358e4c84aec481e67393a3c3, State: Initialized, Role: FOLLOWER
I20260926 07:01:45.046816 2499 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } }
I20260926 07:01:45.046902 2499 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:45.046940 2499 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:45.046993 2499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:45.048117 2499 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } }
I20260926 07:01:45.048204 2499 leader_election.cc:302] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce983183358e4c84aec481e67393a3c3; no voters:
I20260926 07:01:45.048377 2499 leader_election.cc:288] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:45.048429 2504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:45.048566 2504 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [term 1 LEADER]: Becoming Leader. State: Replica: ce983183358e4c84aec481e67393a3c3, State: Running, Role: LEADER
I20260926 07:01:45.048637 2499 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:45.048655 2504 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } }
I20260926 07:01:45.048990 2505 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce983183358e4c84aec481e67393a3c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } } }
I20260926 07:01:45.049094 2505 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:45.049329 2506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce983183358e4c84aec481e67393a3c3. Latest consensus state: current_term: 1 leader_uuid: "ce983183358e4c84aec481e67393a3c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce983183358e4c84aec481e67393a3c3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46101 } } }
I20260926 07:01:45.049686 2510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:45.049750 2506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:45.050191 2510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:45.051983 2510 catalog_manager.cc:1347] Generated new cluster ID: e892b6da4a9e4131995a47ced65ce7b9
I20260926 07:01:45.052032 2510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:45.057711 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.046745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.1.217.126:46101)
0926 07:01:45.046928 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.046938 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.046971 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:01:45.047096 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.047115 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.047157 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.047271 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.048046 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.049440 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.050519 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.051077 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.051211 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.053992 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.053996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.053996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.054008 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.056029 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.056410 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.056413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.056414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.056567 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.056790 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.056791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.056791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.057105 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.057259 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.057358 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.057493 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260926 07:01:45.076973 2510 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:45.077875 2510 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:45.087630 2510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce983183358e4c84aec481e67393a3c3: Generated new TSK 0
I20260926 07:01:45.087836 2510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:45.117957 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:45.199594 2525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:45.199796 2525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:45.199819 2525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:45.200899 2525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:45.200932 2525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:45.200958 2525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:45.202427 2525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:45.202606 2525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:45.202766 2525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:45.207468 2531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.207515 2533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.207581 2530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:45.207935 2525 server_base.cc:1034] running on GCE node
I20260926 07:01:45.208137 2525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:45.208328 2525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:45.209486 2525 hybrid_clock.cc:648] HybridClock initialized: now 1790406105209471 us; error 45 us; skew 500 ppm
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:45.212100 2525 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:45.212584 2525 webserver.cc:460] Webserver started at http://127.1.217.65:43465/ using document root <none> and password file <none>
I20260926 07:01:45.212864 2525 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:45.212932 2525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:45.213078 2525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:45.214121 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "484799cf36fc42f78a98b832eb27a33a"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "f1d5beb920a9a7e05214e33dad4c3605"
server_key_iv: "92a0cd078f0dfe782a19316ee966adbd"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.214490 2525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "484799cf36fc42f78a98b832eb27a33a"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "f1d5beb920a9a7e05214e33dad4c3605"
server_key_iv: "92a0cd078f0dfe782a19316ee966adbd"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.215837 2525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:45.216571 2540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.216749 2525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:45.216871 2525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "484799cf36fc42f78a98b832eb27a33a"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "f1d5beb920a9a7e05214e33dad4c3605"
server_key_iv: "92a0cd078f0dfe782a19316ee966adbd"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.216962 2525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:45.238422 2525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:45.240859 2525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:45.241024 2525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:45.241263 2525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:45.241575 2525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:45.241627 2525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.241669 2525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:45.241696 2525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.247589 2525 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41967
I20260926 07:01:45.247634 2653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41967 every 8 connection(s)
I20260926 07:01:45.247941 2525 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:45.252725 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2525
I20260926 07:01:45.252861 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:45.253041 1893 external_mini_cluster.cc:1428] Setting key dbff94930a838dca783ec91787661c2f
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:45.257825 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.248663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:35187 (local address 127.1.217.126:46101)
0926 07:01:45.248747 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.248749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.248937 (+ 188us) server_negotiation.cc:407] Connection header received
0926 07:01:45.250031 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.250032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.250045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.250108 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.250804 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.251311 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.252310 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.252554 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.252628 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.255078 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.255080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.255080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.255087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.256388 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.256862 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.256864 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.256864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.256926 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.257181 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.257182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.257182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.257296 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.257367 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.257650 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.257735 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:01:45.258044 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.248637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46101 (local address 127.1.217.65:35187)
0926 07:01:45.248809 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.248850 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.249929 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.250194 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.250199 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.250493 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:45.250707 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.250712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.251418 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.251418 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.252222 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.252224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.252636 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.252637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.252733 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.253337 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:45.253346 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:45.254965 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:45.256482 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.256488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.256490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.256724 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.256994 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.256995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.256995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.257108 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.257368 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:45.257375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:45.257510 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.257617 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.257772 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260926 07:01:45.258558 2654 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46101
I20260926 07:01:45.258652 2654 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:45.258850 2654 heartbeater.cc:511] Master 127.1.217.126:46101 requested a full tablet report, sending...
I20260926 07:01:45.259397 2463 ts_manager.cc:194] Registered new tserver with Master: 484799cf36fc42f78a98b832eb27a33a (127.1.217.65:41967)
I20260926 07:01:45.260350 2463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:35187
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:45.279209 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:45.358616 2659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:45.358822 2659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:45.358845 2659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:45.359918 2659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:45.359952 2659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:45.359977 2659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:45.361373 2659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:45.361560 2659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:45.361735 2659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:45.366824 2665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:45.366915 2659 server_base.cc:1034] running on GCE node
W20260926 07:01:45.367030 2667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.366832 2664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:45.367281 2659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:45.367481 2659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:45.368678 2659 hybrid_clock.cc:648] HybridClock initialized: now 1790406105368672 us; error 37 us; skew 500 ppm
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:45.371204 2659 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:45.371644 2659 webserver.cc:460] Webserver started at http://127.1.217.66:41091/ using document root <none> and password file <none>
I20260926 07:01:45.371861 2659 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:45.371905 2659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:45.372068 2659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:45.373224 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "7ec6cd11ffb740ab930da1b71b0781e0"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "2b09235f3c804faa66029f1ad924aad8"
server_key_iv: "3339efab89f938403bf96c46cf992bc9"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.373572 2659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "7ec6cd11ffb740ab930da1b71b0781e0"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "2b09235f3c804faa66029f1ad924aad8"
server_key_iv: "3339efab89f938403bf96c46cf992bc9"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.374850 2659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:45.375571 2674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.375788 2659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:45.375886 2659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "7ec6cd11ffb740ab930da1b71b0781e0"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "2b09235f3c804faa66029f1ad924aad8"
server_key_iv: "3339efab89f938403bf96c46cf992bc9"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.375967 2659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:45.394286 2659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:45.396957 2659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:45.397138 2659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:45.397362 2659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:45.397675 2659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:45.397727 2659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.397766 2659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:45.397794 2659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.403389 2659 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:33073
I20260926 07:01:45.403489 2787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:33073 every 8 connection(s)
I20260926 07:01:45.403805 2659 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:45.412463 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.404582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:51661 (local address 127.1.217.126:46101)
0926 07:01:45.404626 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.404628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.404876 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:01:45.405802 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.405803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.405813 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.405872 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.406490 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.406994 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.407936 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.408122 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.408176 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.410201 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.410203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.410203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.410210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.411388 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.411740 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.411741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.411741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.411787 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.411987 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.411987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.411987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.412079 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.412127 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.412342 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.412399 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:45.412715 2790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.404540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46101 (local address 127.1.217.66:51661)
0926 07:01:45.404712 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.404754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.405704 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.405927 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.405930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.406182 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:45.406386 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.406390 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.407066 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.407067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.407860 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.407861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.408191 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.408192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.408292 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.408736 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:45.408744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:45.410120 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:45.411454 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.411460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.411461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.411667 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.411841 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.411842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.411842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.411933 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.412131 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:45.412134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:45.412239 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.412318 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.412444 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 07:01:45.413255 2788 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46101
I20260926 07:01:45.413362 2788 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:45.413609 2788 heartbeater.cc:511] Master 127.1.217.126:46101 requested a full tablet report, sending...
I20260926 07:01:45.413862 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2659
I20260926 07:01:45.413933 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:45.414114 1893 external_mini_cluster.cc:1428] Setting key 0123097516aa65804c28b530f30e80f2
I20260926 07:01:45.414541 2462 ts_manager.cc:194] Registered new tserver with Master: 7ec6cd11ffb740ab930da1b71b0781e0 (127.1.217.66:33073)
I20260926 07:01:45.414989 2462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:51661
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:45.439254 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:45.517619 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:45.517856 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:45.517894 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:45.519048 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:45.519119 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:45.519165 2793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:45.520676 2793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46101
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:45.520995 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:45.521229 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:45.526168 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.526218 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:45.526232 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:45.526453 2793 server_base.cc:1034] running on GCE node
I20260926 07:01:45.526679 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:45.526933 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:45.528090 2793 hybrid_clock.cc:648] HybridClock initialized: now 1790406105528071 us; error 34 us; skew 500 ppm
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:45.530624 2793 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:45.531088 2793 webserver.cc:460] Webserver started at http://127.1.217.67:44425/ using document root <none> and password file <none>
I20260926 07:01:45.531327 2793 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:45.531370 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:45.531510 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:45.532615 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "a709ad5b495b45d99e684b27f2b72f22"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "e40541e23ce942db1530639c20598ec5"
server_key_iv: "76db6aca221eae6762cdd97663c228ef"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.533020 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "a709ad5b495b45d99e684b27f2b72f22"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "e40541e23ce942db1530639c20598ec5"
server_key_iv: "76db6aca221eae6762cdd97663c228ef"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.534300 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:45.534983 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.535187 2793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:45.535274 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "a709ad5b495b45d99e684b27f2b72f22"
format_stamp: "Formatted at 2026-09-26 07:01:45 on dist-test-slave-t1mc"
server_key: "e40541e23ce942db1530639c20598ec5"
server_key_iv: "76db6aca221eae6762cdd97663c228ef"
server_key_version: "encryptionkey@0"
I20260926 07:01:45.535339 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:45.561110 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:45.563390 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:45.563550 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:45.563776 2793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:45.564080 2793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:45.564113 2793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.564167 2793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:45.564194 2793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:45.569872 2793 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:32791
I20260926 07:01:45.569945 2921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:32791 every 8 connection(s)
I20260926 07:01:45.570267 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:45.573949 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2793
I20260926 07:01:45.574057 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:45.574210 1893 external_mini_cluster.cc:1428] Setting key ce2f6bc816c368f13f1a49b60a73a4ef
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:45.579041 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.571012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:50729 (local address 127.1.217.126:46101)
0926 07:01:45.571078 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.571080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.571242 (+ 162us) server_negotiation.cc:407] Connection header received
0926 07:01:45.572131 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.572132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.572142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.572208 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.572749 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.573275 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.574302 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.574563 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.574652 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.576667 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.576668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.576669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.576675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.577874 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.578222 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.578223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.578223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.578270 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.578467 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.578468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.578468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.578643 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.578702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.578918 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.578977 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:01:45.579196 2924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.570981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46101 (local address 127.1.217.67:50729)
0926 07:01:45.571123 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.571163 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.572047 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.572260 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.572264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.572499 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:45.572682 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.572686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.573350 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.573351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.574214 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.574216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.574671 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.574672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.574826 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.575284 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:45.575291 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:45.576591 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:45.577945 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.577950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.577952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.578158 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.578326 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:45.578326 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:45.578326 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:45.578416 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:45.578696 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:45.578701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:45.578807 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.578892 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.579002 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260926 07:01:45.579741 2922 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46101
I20260926 07:01:45.579850 2922 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:45.580017 2922 heartbeater.cc:511] Master 127.1.217.126:46101 requested a full tablet report, sending...
I20260926 07:01:45.580391 2462 ts_manager.cc:194] Registered new tserver with Master: a709ad5b495b45d99e684b27f2b72f22 (127.1.217.67:32791)
I20260926 07:01:45.580714 2462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:50729
I20260926 07:01:45.585608 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:45.592537 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.587088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40724 (local address 127.1.217.126:46101)
0926 07:01:45.587176 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.587178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.587228 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:01:45.587325 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.587326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.587336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.587363 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.587944 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.588401 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.589149 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.589323 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.589375 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.590485 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.590486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.590486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.590491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.591593 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.591888 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.591889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.591889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.591933 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.592088 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.592088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.592088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.592211 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.592281 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.592369 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.592421 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:01:45.594595 2462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:45.596052 2462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:45.602629 2934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41967 (local address 127.0.0.1:57472)
0926 07:01:45.598477 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.598510 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.598565 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.598876 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.598879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.598885 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:45.598972 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.598973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600272 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.600273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.601405 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.601407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602330 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.602414 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602419 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.602466 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.602507 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":45,"thread_start_us":48,"threads_started":1}
I20260926 07:01:45.602793 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.1.217.66:33073)
0926 07:01:45.598640 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.598645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.599136 (+ 491us) server_negotiation.cc:407] Connection header received
0926 07:01:45.599232 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.599235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.599252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.599328 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:45.599449 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600115 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.601722 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602167 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602315 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.602667 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.602708 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260926 07:01:45.602864 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.1.217.67:32791)
0926 07:01:45.598725 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.598728 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.598737 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:45.598775 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.598776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.598791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.598864 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:45.599060 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600181 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.601704 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602362 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602450 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602500 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.602601 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.602639 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":81,"threads_started":1}
I20260926 07:01:45.602908 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.1.217.65:41967)
0926 07:01:45.598668 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.598672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.598683 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:45.598727 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.598729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.598746 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.598829 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:45.599093 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600154 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.601487 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602169 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602239 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602296 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.602787 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.602829 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260926 07:01:45.602716 2933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32791 (local address 127.0.0.1:45972)
0926 07:01:45.598367 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.598404 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.598466 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.598876 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.598879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.598882 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:45.598970 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.598972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600324 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.600324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.601625 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.601628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602456 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602456 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.602529 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602533 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.602586 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.602623 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":101,"thread_start_us":75,"threads_started":1}
I20260926 07:01:45.602631 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.598229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:33073 (local address 127.0.0.1:56512)
0926 07:01:45.598992 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.599022 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.599077 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.599324 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.599324 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.599326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:45.599369 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.599371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.600233 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.600234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.601630 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.601631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.602265 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.602266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.602339 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.602352 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.602446 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.602497 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":73,"thread_start_us":85,"threads_started":1}
I20260926 07:01:45.603703 2856 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea573c2c684fe78d572b63d19e9dbb (DEFAULT_TABLE table=test-table [id=6abfbe7dd0944449a2bc416baf4c7075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:45.603955 2856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea573c2c684fe78d572b63d19e9dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:45.603936 2722 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea573c2c684fe78d572b63d19e9dbb (DEFAULT_TABLE table=test-table [id=6abfbe7dd0944449a2bc416baf4c7075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:45.604195 2722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea573c2c684fe78d572b63d19e9dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:45.606544 2939 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: Bootstrap starting.
I20260926 07:01:45.606951 2588 tablet_service.cc:1467] Processing CreateTablet for tablet e2ea573c2c684fe78d572b63d19e9dbb (DEFAULT_TABLE table=test-table [id=6abfbe7dd0944449a2bc416baf4c7075]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:45.607180 2588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ea573c2c684fe78d572b63d19e9dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:45.607182 2939 tablet_bootstrap.cc:654] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:45.607604 2939 log.cc:824] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:45.608244 2939 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: No bootstrap required, opened a new log
I20260926 07:01:45.608285 2939 ts_tablet_manager.cc:1397] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:45.609077 2940 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Bootstrap starting.
I20260926 07:01:45.609853 2940 tablet_bootstrap.cc:654] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:45.610236 2939 raft_consensus.cc:348] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.610404 2940 log.cc:824] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:45.610425 2939 raft_consensus.cc:374] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:45.610462 2939 raft_consensus.cc:729] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a709ad5b495b45d99e684b27f2b72f22, State: Initialized, Role: FOLLOWER
I20260926 07:01:45.610587 2939 consensus_queue.cc:260] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.610870 2939 ts_tablet_manager.cc:1428] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:45.610893 2922 heartbeater.cc:503] Master 127.1.217.126:46101 was elected leader, sending a full tablet report...
I20260926 07:01:45.611054 2942 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: Bootstrap starting.
I20260926 07:01:45.611114 2940 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: No bootstrap required, opened a new log
I20260926 07:01:45.611171 2940 ts_tablet_manager.cc:1397] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:45.611654 2942 tablet_bootstrap.cc:654] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:45.612193 2942 log.cc:824] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:45.612936 2942 tablet_bootstrap.cc:492] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: No bootstrap required, opened a new log
I20260926 07:01:45.613009 2942 ts_tablet_manager.cc:1397] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:01:45.613025 2940 raft_consensus.cc:348] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.613209 2940 raft_consensus.cc:374] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:45.613255 2940 raft_consensus.cc:729] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec6cd11ffb740ab930da1b71b0781e0, State: Initialized, Role: FOLLOWER
I20260926 07:01:45.613356 2940 consensus_queue.cc:260] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.613642 2940 ts_tablet_manager.cc:1428] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 07:01:45.613662 2788 heartbeater.cc:503] Master 127.1.217.126:46101 was elected leader, sending a full tablet report...
I20260926 07:01:45.614897 2942 raft_consensus.cc:348] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.615047 2942 raft_consensus.cc:374] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:45.615092 2942 raft_consensus.cc:729] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 484799cf36fc42f78a98b832eb27a33a, State: Initialized, Role: FOLLOWER
I20260926 07:01:45.615196 2942 consensus_queue.cc:260] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.615444 2942 ts_tablet_manager.cc:1428] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:01:45.615468 2654 heartbeater.cc:503] Master 127.1.217.126:46101 was elected leader, sending a full tablet report...
I20260926 07:01:45.633725 2947 raft_consensus.cc:482] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:45.633872 2947 raft_consensus.cc:504] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.634274 2947 leader_election.cc:288] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a709ad5b495b45d99e684b27f2b72f22 (127.1.217.67:32791), 7ec6cd11ffb740ab930da1b71b0781e0 (127.1.217.66:33073)
I20260926 07:01:45.638159 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.634467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:40779 (local address 127.1.217.67:32791)
0926 07:01:45.634545 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.634547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.634558 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:01:45.634593 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.634594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.634603 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.634662 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:45.634865 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.635631 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637241 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.637837 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637900 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.637907 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.638050 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.638090 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 07:01:45.638159 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.634539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:57697 (local address 127.1.217.66:33073)
0926 07:01:45.634582 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.634584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.634717 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:01:45.634746 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.634747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.634756 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.634805 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:45.634956 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.635632 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637242 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.637792 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637871 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.637882 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.638054 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.638090 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:45.638185 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.634418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32791 (local address 127.1.217.65:40779)
0926 07:01:45.634476 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.634489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.634537 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.634687 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.634688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.634690 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:45.634769 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.634771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.635727 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.635728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.637130 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.637133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.637901 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.637978 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.637981 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.638032 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.638066 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":124}
I20260926 07:01:45.638191 2948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.634451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:33073 (local address 127.1.217.65:57697)
0926 07:01:45.634628 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:45.634635 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:45.634703 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:45.634828 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:45.634829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:45.634831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:45.634889 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.634890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.635727 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.635728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.637130 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:45.637133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.637885 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.637886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:45.637959 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.637964 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:45.638027 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:01:45.638066 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":37,"thread_start_us":103,"threads_started":1}
I20260926 07:01:45.638584 2876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea573c2c684fe78d572b63d19e9dbb" candidate_uuid: "484799cf36fc42f78a98b832eb27a33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a709ad5b495b45d99e684b27f2b72f22" is_pre_election: true
I20260926 07:01:45.638741 2876 raft_consensus.cc:2454] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484799cf36fc42f78a98b832eb27a33a in term 0.
I20260926 07:01:45.638752 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea573c2c684fe78d572b63d19e9dbb" candidate_uuid: "484799cf36fc42f78a98b832eb27a33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" is_pre_election: true
I20260926 07:01:45.638897 2742 raft_consensus.cc:2454] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 484799cf36fc42f78a98b832eb27a33a in term 0.
I20260926 07:01:45.639011 2542 leader_election.cc:302] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484799cf36fc42f78a98b832eb27a33a, a709ad5b495b45d99e684b27f2b72f22; no voters:
I20260926 07:01:45.639137 2947 raft_consensus.cc:2781] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:45.639199 2947 raft_consensus.cc:482] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:45.639221 2947 raft_consensus.cc:3037] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:45.639832 2947 raft_consensus.cc:504] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.639966 2947 leader_election.cc:288] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [CANDIDATE]: Term 1 election: Requested vote from peers a709ad5b495b45d99e684b27f2b72f22 (127.1.217.67:32791), 7ec6cd11ffb740ab930da1b71b0781e0 (127.1.217.66:33073)
I20260926 07:01:45.640115 2742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea573c2c684fe78d572b63d19e9dbb" candidate_uuid: "484799cf36fc42f78a98b832eb27a33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ec6cd11ffb740ab930da1b71b0781e0"
I20260926 07:01:45.640201 2742 raft_consensus.cc:3037] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:45.640995 2742 raft_consensus.cc:2454] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484799cf36fc42f78a98b832eb27a33a in term 1.
I20260926 07:01:45.641134 2876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ea573c2c684fe78d572b63d19e9dbb" candidate_uuid: "484799cf36fc42f78a98b832eb27a33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a709ad5b495b45d99e684b27f2b72f22"
I20260926 07:01:45.641155 2544 leader_election.cc:302] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 484799cf36fc42f78a98b832eb27a33a, 7ec6cd11ffb740ab930da1b71b0781e0; no voters:
I20260926 07:01:45.641232 2876 raft_consensus.cc:3037] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:45.641269 2947 raft_consensus.cc:2781] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:45.641417 2947 raft_consensus.cc:686] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [term 1 LEADER]: Becoming Leader. State: Replica: 484799cf36fc42f78a98b832eb27a33a, State: Running, Role: LEADER
I20260926 07:01:45.641520 2947 consensus_queue.cc:237] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } }
I20260926 07:01:45.642102 2876 raft_consensus.cc:2454] T e2ea573c2c684fe78d572b63d19e9dbb P a709ad5b495b45d99e684b27f2b72f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 484799cf36fc42f78a98b832eb27a33a in term 1.
I20260926 07:01:45.642325 2463 catalog_manager.cc:5495] T e2ea573c2c684fe78d572b63d19e9dbb P 484799cf36fc42f78a98b832eb27a33a reported cstate change: term changed from 0 to 1, leader changed from <none> to 484799cf36fc42f78a98b832eb27a33a (127.1.217.65). New cstate: current_term: 1 leader_uuid: "484799cf36fc42f78a98b832eb27a33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "484799cf36fc42f78a98b832eb27a33a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a709ad5b495b45d99e684b27f2b72f22" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ec6cd11ffb740ab930da1b71b0781e0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33073 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:45.651288 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.644288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.1.217.65:41967)
0926 07:01:45.644346 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.644348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.644467 (+ 119us) server_negotiation.cc:407] Connection header received
0926 07:01:45.644523 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.644523 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.644531 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.644585 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.645181 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.645618 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.646295 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.646496 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.646559 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.648731 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.648738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.648738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.648745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.650385 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.650708 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.650710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.650711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.650761 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.650886 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.650887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.650887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.651019 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.651138 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.651170 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.651226 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:01:45.654368 2789 tablet.cc:2369] T e2ea573c2c684fe78d572b63d19e9dbb P 7ec6cd11ffb740ab930da1b71b0781e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:45.673516 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.666428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.1.217.65:41967)
0926 07:01:45.666510 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.666513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.666627 (+ 114us) server_negotiation.cc:407] Connection header received
0926 07:01:45.666735 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.666736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.666746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.666817 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.667376 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.667851 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.668606 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.668865 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.668967 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.671104 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.671106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.671106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.671115 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.672269 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.672633 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.672635 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.672635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.672692 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.672923 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.672924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.672924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.673035 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.673140 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.673331 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.673411 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:45.696024 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:45.689118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.1.217.65:41967)
0926 07:01:45.689182 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:45.689184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:45.689294 (+ 110us) server_negotiation.cc:407] Connection header received
0926 07:01:45.689341 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:45.689342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:45.689350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:45.689406 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:45.689946 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.690400 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.691364 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:45.691554 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:45.691607 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:45.693704 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:45.693706 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:45.693706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:45.693714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:45.694895 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.695269 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.695271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.695271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.695328 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:45.695536 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:45.695537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:45.695537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:45.695631 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:45.695728 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:45.695846 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:01:45.695921 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 07:01:45.696475 2568 tablet_service.cc:2908] Rejecting scan request for tablet e2ea573c2c684fe78d572b63d19e9dbb: Uninitialized: clean time has not yet been initialized
I20260926 07:01:45.697330 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2525
I20260926 07:01:45.702344 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2659
I20260926 07:01:45.707000 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2793
I20260926 07:01:45.711150 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2431
2026-09-26T07:01:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1946 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2963](info): set up 2 sockets
Sep 26 07:01:45 dist-test-slave-t1mc krb5kdc[2963](info): commencing operation
krb5kdc: starting...
W20260926 07:01:46.740176 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:46 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:46.856127 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33745
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:46.935077 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:46.935313 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:46.935353 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:46.936479 2976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:46.936550 2976 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:01:46.936573 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:46.936594 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:46.936621 2976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:46.936645 2976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:46.938104 2976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:46.938370 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:46.938588 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:46.944495 2981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:46.948024 2982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:46.948051 2984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:46.948752 2976 server_base.cc:1034] running on GCE node
I20260926 07:01:46.949098 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:46.949481 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:46.950641 2976 hybrid_clock.cc:648] HybridClock initialized: now 1790406106950620 us; error 38 us; skew 500 ppm
Sep 26 07:01:46 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:46.958494 2976 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:46.958973 2976 webserver.cc:460] Webserver started at http://127.1.217.126:37841/ using document root <none> and password file <none>
I20260926 07:01:46.959218 2976 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:46.959291 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:46.959460 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:46.960700 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d258f74226db4bc9b6adf37fe5bd93dd"
format_stamp: "Formatted at 2026-09-26 07:01:46 on dist-test-slave-t1mc"
server_key: "320c3e6c4834497e14b0bae6bdf6fc85"
server_key_iv: "c33e8c508d2900fb1702704ab2718902"
server_key_version: "encryptionkey@0"
I20260926 07:01:46.961151 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d258f74226db4bc9b6adf37fe5bd93dd"
format_stamp: "Formatted at 2026-09-26 07:01:46 on dist-test-slave-t1mc"
server_key: "320c3e6c4834497e14b0bae6bdf6fc85"
server_key_iv: "c33e8c508d2900fb1702704ab2718902"
server_key_version: "encryptionkey@0"
I20260926 07:01:46.962560 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:46.963304 2991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:46.963482 2976 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:46.963599 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d258f74226db4bc9b6adf37fe5bd93dd"
format_stamp: "Formatted at 2026-09-26 07:01:46 on dist-test-slave-t1mc"
server_key: "320c3e6c4834497e14b0bae6bdf6fc85"
server_key_iv: "c33e8c508d2900fb1702704ab2718902"
server_key_version: "encryptionkey@0"
I20260926 07:01:46.963685 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:46.998370 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:47.000404 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:47.000581 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:47.005681 2976 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33745
I20260926 07:01:47.005733 3053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33745 every 8 connection(s)
I20260926 07:01:47.006108 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:47.006783 3054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.008947 3054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Bootstrap starting.
I20260926 07:01:47.009483 3054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.009716 3054 log.cc:824] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:47.010341 3054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: No bootstrap required, opened a new log
I20260926 07:01:47.011124 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 2976
I20260926 07:01:47.011226 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:47.011404 1893 external_mini_cluster.cc:1428] Setting key 18261446621e63543e9a90cc97dcd6af
I20260926 07:01:47.012964 3054 raft_consensus.cc:348] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } }
I20260926 07:01:47.013149 3054 raft_consensus.cc:374] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.013190 3054 raft_consensus.cc:729] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d258f74226db4bc9b6adf37fe5bd93dd, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.013319 3054 consensus_queue.cc:260] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } }
I20260926 07:01:47.013398 3054 raft_consensus.cc:388] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:47.013422 3054 raft_consensus.cc:482] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:47.013489 3054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.014113 3054 raft_consensus.cc:504] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } }
I20260926 07:01:47.014214 3054 leader_election.cc:302] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d258f74226db4bc9b6adf37fe5bd93dd; no voters:
I20260926 07:01:47.014360 3054 leader_election.cc:288] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:47.014434 3059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:47.014590 3059 raft_consensus.cc:686] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [term 1 LEADER]: Becoming Leader. State: Replica: d258f74226db4bc9b6adf37fe5bd93dd, State: Running, Role: LEADER
I20260926 07:01:47.014601 3054 sys_catalog.cc:564] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:47.014698 3059 consensus_queue.cc:237] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } }
I20260926 07:01:47.015057 3059 sys_catalog.cc:455] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d258f74226db4bc9b6adf37fe5bd93dd. Latest consensus state: current_term: 1 leader_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } } }
I20260926 07:01:47.015072 3060 sys_catalog.cc:455] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d258f74226db4bc9b6adf37fe5bd93dd" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33745 } } }
I20260926 07:01:47.015167 3060 sys_catalog.cc:458] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:47.015367 3059 sys_catalog.cc:458] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:47.015770 3065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:47.016467 3065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:47.018177 3065 catalog_manager.cc:1347] Generated new cluster ID: 2719fd11e5dd4cce8704be67f70dfcff
I20260926 07:01:47.018227 3065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.024287 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.011928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.1.217.126:33745)
0926 07:01:47.012145 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.012157 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.012201 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:01:47.012922 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.012931 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.012995 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.013148 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.013919 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.015591 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.016714 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.017379 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.017497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.020388 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.020391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.020392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.020404 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.022600 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.023010 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.023014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.023014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.023135 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.023345 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.023346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.023347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.023671 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.023826 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.023937 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.024077 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":96,"threads_started":1}
I20260926 07:01:47.043074 3065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:47.043828 3065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:47.051981 3065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Generated new TSK 0
I20260926 07:01:47.052169 3065 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:47.076275 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33745
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:47.156533 3080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:47.156734 3080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:47.156757 3080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:47.158006 3080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:47.158047 3080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:47.158068 3080 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:47.158099 3080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:47.159559 3080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:47.159793 3080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:47.160019 3080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:47.164943 3085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.164956 3086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.164956 3088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:47.165309 3080 server_base.cc:1034] running on GCE node
I20260926 07:01:47.165515 3080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:47.165740 3080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:47.166896 3080 hybrid_clock.cc:648] HybridClock initialized: now 1790406107166873 us; error 42 us; skew 500 ppm
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:47.169502 3080 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:47.169958 3080 webserver.cc:460] Webserver started at http://127.1.217.65:40795/ using document root <none> and password file <none>
I20260926 07:01:47.170179 3080 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:47.170246 3080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:47.170380 3080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:47.171375 3080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "42e688ca81a84f7c9ded1d25484966c1"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "0361ee0b6fee89aeba5565ffe53213cc"
server_key_iv: "7bbcc729a58eaaaaa4a51de8be1a2ace"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.171725 3080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "42e688ca81a84f7c9ded1d25484966c1"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "0361ee0b6fee89aeba5565ffe53213cc"
server_key_iv: "7bbcc729a58eaaaaa4a51de8be1a2ace"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.173096 3080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:47.173883 3095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.174077 3080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:47.174178 3080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "42e688ca81a84f7c9ded1d25484966c1"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "0361ee0b6fee89aeba5565ffe53213cc"
server_key_iv: "7bbcc729a58eaaaaa4a51de8be1a2ace"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.174249 3080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:47.204006 3080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:47.206593 3080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:47.206756 3080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:47.207388 3080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:47.207439 3080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.207473 3080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:47.207491 3080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.214550 3080 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43045
I20260926 07:01:47.214895 3080 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:47.214881 3210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43045 every 8 connection(s)
I20260926 07:01:47.217375 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.207846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:38501 (local address 127.1.217.126:33745)
0926 07:01:47.207915 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.207918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.208101 (+ 183us) server_negotiation.cc:407] Connection header received
0926 07:01:47.208566 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.208567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.208580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.208671 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.209397 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.210245 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.211403 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.211608 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.211687 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.214306 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.214308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.214308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.214316 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.215998 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.216397 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.216399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.216399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.216453 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.216704 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.216704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.216705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.216862 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.216939 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.217203 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.217283 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:01:47.217559 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.207812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.65:38501)
0926 07:01:47.207962 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.208009 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.208475 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.208772 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.208792 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.209105 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.209297 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.209302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.210372 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.210373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.211302 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.211305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.211692 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.211693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.211819 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.212289 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.212296 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.214201 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.216088 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.216094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.216097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.216309 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.216527 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.216528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.216528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.216636 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.216930 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.216936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.217073 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.217181 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.217329 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260926 07:01:47.217998 3211 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33745
I20260926 07:01:47.218077 3211 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:47.218261 3211 heartbeater.cc:511] Master 127.1.217.126:33745 requested a full tablet report, sending...
I20260926 07:01:47.219110 3008 ts_manager.cc:194] Registered new tserver with Master: 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045)
I20260926 07:01:47.219931 3008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:38501
I20260926 07:01:47.221158 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3080
I20260926 07:01:47.221300 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:47.221518 1893 external_mini_cluster.cc:1428] Setting key 294bc42145c4a384907f4fd5cf1839e6
I20260926 07:01:47.224551 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.219057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:57459 (local address 127.1.217.126:33745)
0926 07:01:47.219095 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.219097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.219167 (+ 70us) server_negotiation.cc:407] Connection header received
0926 07:01:47.219241 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.219242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.219251 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.219306 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.219635 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.220153 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.220956 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.221164 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.221256 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.222250 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.222251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.222252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.222258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.223447 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.223791 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.223792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.223792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.223843 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.224048 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.224049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.224049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.224158 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.224235 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.224392 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.224459 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:47.224571 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.218966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.65:57459)
0926 07:01:47.219108 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.219119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.219183 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.219315 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.219316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.219507 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.219567 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.219569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.220244 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.220245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.220884 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.220887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.221257 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.221258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.221326 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.221772 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.221776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.222157 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.223545 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.223546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.223546 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.223709 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.223915 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.223916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.223917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.223978 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.224228 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.224229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.224269 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.224368 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.224459 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:47.247076 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33745
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:47.328799 3220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:47.329007 3220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:47.329032 3220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:47.330219 3220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:47.330263 3220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:47.330287 3220 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:47.330307 3220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:47.331854 3220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:47.332105 3220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:47.332333 3220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:47.337461 3228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.337509 3226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.337505 3225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:47.337656 3220 server_base.cc:1034] running on GCE node
I20260926 07:01:47.337891 3220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:47.338083 3220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:47.339218 3220 hybrid_clock.cc:648] HybridClock initialized: now 1790406107339201 us; error 32 us; skew 500 ppm
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:47.341818 3220 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:47.342252 3220 webserver.cc:460] Webserver started at http://127.1.217.66:38175/ using document root <none> and password file <none>
I20260926 07:01:47.342497 3220 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:47.342563 3220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:47.342691 3220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:47.343699 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "b331f9f7fc9b4eefa688da729bfa4e36"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "cce9e0563762042134d2ee5f453597d8"
server_key_iv: "415ed0cfd9e56977fcf714a527ac2e48"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.344064 3220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "b331f9f7fc9b4eefa688da729bfa4e36"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "cce9e0563762042134d2ee5f453597d8"
server_key_iv: "415ed0cfd9e56977fcf714a527ac2e48"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.345489 3220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 07:01:47.346246 3235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.346479 3220 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:47.346575 3220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "b331f9f7fc9b4eefa688da729bfa4e36"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "cce9e0563762042134d2ee5f453597d8"
server_key_iv: "415ed0cfd9e56977fcf714a527ac2e48"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.346652 3220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:47.361322 3220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:47.363265 3220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:47.363445 3220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:47.363972 3220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:47.364028 3220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.364068 3220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:47.364090 3220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.371188 3220 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:46751
I20260926 07:01:47.371625 3220 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:47.371764 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3220
I20260926 07:01:47.371860 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:47.372043 1893 external_mini_cluster.cc:1428] Setting key e6c3ca7c1d482e0b1ef8c4756f1fbdf2
I20260926 07:01:47.372014 3350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:46751 every 8 connection(s)
I20260926 07:01:47.374451 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.364357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:51249 (local address 127.1.217.126:33745)
0926 07:01:47.364432 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.364435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.364682 (+ 247us) server_negotiation.cc:407] Connection header received
0926 07:01:47.365201 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.365203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.365217 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.365293 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.365970 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.366856 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.368451 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.368788 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.368851 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.371214 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.371215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.371215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.371224 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.372985 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.373458 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.373459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.373459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.373522 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.373754 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.373755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.373755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.373901 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.373968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.374319 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.374389 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 07:01:47.374758 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.364393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.66:51249)
0926 07:01:47.364542 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.364582 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.365105 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.365387 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.365393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.365678 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.365878 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.365883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.366952 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.366953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.368371 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.368374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.368875 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.368876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.369028 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.369546 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.369554 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.371099 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.373098 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.373104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.373105 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.373379 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.373589 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.373590 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.373590 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.373688 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.373963 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.373967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.374168 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.374295 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.374512 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260926 07:01:47.375276 3351 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33745
I20260926 07:01:47.375350 3351 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:47.375519 3351 heartbeater.cc:511] Master 127.1.217.126:33745 requested a full tablet report, sending...
I20260926 07:01:47.375826 3008 ts_manager.cc:194] Registered new tserver with Master: b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66:46751)
I20260926 07:01:47.376279 3008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:51249
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
I20260926 07:01:47.386265 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.376347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:34521 (local address 127.1.217.126:33745)
0926 07:01:47.376427 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.376429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.380588 (+ 4159us) server_negotiation.cc:407] Connection header received
0926 07:01:47.380660 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.380661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.380671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.380736 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.381076 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.381598 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.382299 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.382501 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.382590 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.383727 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.383728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.383728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.383734 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.385144 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.385522 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.385522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.385523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.385571 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.385764 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.385764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.385765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.385902 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.385953 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.386134 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.386192 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:01:47.386358 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.376301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.66:34521)
0926 07:01:47.380507 (+ 4206us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.380518 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.380600 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.380754 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.380756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.380928 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.380987 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.380988 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.381698 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.381699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.382214 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.382216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.382592 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.382593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.382666 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.383183 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.383188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.383632 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.385222 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.385223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.385223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.385448 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.385634 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.385635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.385636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.385706 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.385966 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.385967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.386027 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.386105 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.386219 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4183,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:47.398947 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33745
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:47.479952 3360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:47.480159 3360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:47.480182 3360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:47.481294 3360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:47.481338 3360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:47.481357 3360 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:47.481374 3360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:47.482731 3360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:33745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:47.482965 3360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:47.483148 3360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:47.487954 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.487965 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:47.487986 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:47.488413 3360 server_base.cc:1034] running on GCE node
I20260926 07:01:47.488659 3360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:47.488932 3360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:47.490085 3360 hybrid_clock.cc:648] HybridClock initialized: now 1790406107490069 us; error 29 us; skew 500 ppm
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:47.492604 3360 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:47.493144 3360 webserver.cc:460] Webserver started at http://127.1.217.67:40313/ using document root <none> and password file <none>
I20260926 07:01:47.493386 3360 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:47.493451 3360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:47.493608 3360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:47.494652 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "4240eaaa208deaa6c83b0ca5df642310"
server_key_iv: "f61585cee7ab0d19ba77a246adef9df8"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.495018 3360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "4240eaaa208deaa6c83b0ca5df642310"
server_key_iv: "f61585cee7ab0d19ba77a246adef9df8"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.496321 3360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:47.497023 3375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.497185 3360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:47.497310 3360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
format_stamp: "Formatted at 2026-09-26 07:01:47 on dist-test-slave-t1mc"
server_key: "4240eaaa208deaa6c83b0ca5df642310"
server_key_iv: "f61585cee7ab0d19ba77a246adef9df8"
server_key_version: "encryptionkey@0"
I20260926 07:01:47.497385 3360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:47.507644 3360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:47.509414 3360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:47.509567 3360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:47.510133 3360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:47.510190 3360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.510232 3360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:47.510273 3360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.517304 3360 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:43101
I20260926 07:01:47.517733 3360 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:47.518157 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:43101 every 8 connection(s)
I20260926 07:01:47.519824 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.510586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:51699 (local address 127.1.217.126:33745)
0926 07:01:47.510654 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.510657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.510825 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:01:47.511265 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.511267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.511278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.511363 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.512040 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.512538 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.513720 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.513950 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.514037 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.517184 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.517186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.517186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.517193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.518651 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.519039 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.519039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.519040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.519088 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.519290 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.519290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.519291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.519390 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.519440 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.519684 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.519751 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:01:47.519909 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.510557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.67:51699)
0926 07:01:47.510694 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.510731 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.511176 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.511411 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.511414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.511710 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.511941 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.511953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.512628 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.512629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.513633 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.513636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.514047 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.514048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.514174 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.514665 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.514672 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.517079 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.518721 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.518724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.518726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.518917 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.519146 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.519146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.519147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.519237 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.519444 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.519448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.519549 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.519626 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.519734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260926 07:01:47.520376 3491 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33745
I20260926 07:01:47.520478 3491 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:47.520640 3491 heartbeater.cc:511] Master 127.1.217.126:33745 requested a full tablet report, sending...
I20260926 07:01:47.521054 3008 ts_manager.cc:194] Registered new tserver with Master: d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:47.521446 3008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:51699
I20260926 07:01:47.523608 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3360
I20260926 07:01:47.523712 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:47.523922 1893 external_mini_cluster.cc:1428] Setting key 686ac0800aa7c08ce211268ff54e093a
I20260926 07:01:47.524745 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:47.527001 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.521528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60845 (local address 127.1.217.126:33745)
0926 07:01:47.521569 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.521571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.521671 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:01:47.521716 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.521717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.521728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.521785 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.522108 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.522569 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.523203 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.523389 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.523447 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.524390 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.524391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.524392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.524399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.525974 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.526284 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.526285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.526285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.526355 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.526528 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.526529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.526529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.526666 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.526737 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.526887 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.526941 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:47.527108 3497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.521441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.1.217.67:60845)
0926 07:01:47.521603 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.521614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.521664 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.521788 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.521789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.521982 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.522040 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.522042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.522648 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.522649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.523139 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.523140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.523459 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.523460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.523541 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.523951 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.523956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.524317 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.526052 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.526053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.526053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.526205 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.526421 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.526421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.526421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.526471 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.526738 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.526739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.526789 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.526858 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.526974 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.547912 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.541002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.1.217.126:33745)
0926 07:01:47.541058 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.541060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.541069 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:47.541096 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.541097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.541106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.541175 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.541705 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.542201 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.542951 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.543147 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.543234 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.545426 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.545427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.545427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.545435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.546810 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.547156 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.547157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.547158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.547217 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.547412 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.547413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.547413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.547521 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.547627 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.547730 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.547805 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:01:47.549760 3008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:47.551426 3008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:47.557978 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.553888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46751 (local address 127.0.0.1:35830)
0926 07:01:47.554224 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.554233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.554282 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.554491 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.554491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.554493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.554530 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.554532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555777 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.555778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.556929 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.556930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557646 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.557723 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557737 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.557837 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.557884 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":27,"thread_start_us":50,"threads_started":1}
I20260926 07:01:47.557998 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.554003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35830 (local address 127.1.217.66:46751)
0926 07:01:47.554216 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.554221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.554280 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:01:47.554371 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.554373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.554388 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.554600 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.554655 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555704 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557015 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557546 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557705 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.557855 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.557894 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":82,"threads_started":1}
I20260926 07:01:47.558035 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.553818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.1.217.65:43045)
0926 07:01:47.554021 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.554026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.554128 (+ 102us) server_negotiation.cc:407] Connection header received
0926 07:01:47.554235 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.554238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.554257 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.554341 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.554531 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555653 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.556825 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557571 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557646 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557706 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.557888 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.557934 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260926 07:01:47.558245 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.553701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43045 (local address 127.0.0.1:37038)
0926 07:01:47.553998 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.554038 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.554114 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.554358 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.554361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.554364 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.554437 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.554439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555772 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.555772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.556711 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.556713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557733 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.557798 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557802 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.558118 (+ 316us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.558172 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":20,"thread_start_us":61,"threads_started":1}
I20260926 07:01:47.558413 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.553891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.1.217.67:43101)
0926 07:01:47.554475 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.554479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.554489 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:47.554525 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.554527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.554541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.554784 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.554827 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555471 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.556897 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557578 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557932 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557994 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.558016 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.558243 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":77,"threads_started":1}
I20260926 07:01:47.558696 3285 tablet_service.cc:1467] Processing CreateTablet for tablet 5afe19049971434284d13cd19109dc41 (DEFAULT_TABLE table=test-table [id=8385845da3584aaa90c4affd0260eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.559062 3285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afe19049971434284d13cd19109dc41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.559020 3145 tablet_service.cc:1467] Processing CreateTablet for tablet 5afe19049971434284d13cd19109dc41 (DEFAULT_TABLE table=test-table [id=8385845da3584aaa90c4affd0260eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.559322 3145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afe19049971434284d13cd19109dc41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.560914 3507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.553704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:43101 (local address 127.0.0.1:36760)
0926 07:01:47.554074 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.554084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.554137 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.554638 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.554639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.554641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.554715 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.554717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.555556 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.555557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.556828 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.556830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.557675 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.557676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.557740 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.557744 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.557782 (+ 38us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.557862 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":56,"threads_started":1}
I20260926 07:01:47.561626 3419 tablet_service.cc:1467] Processing CreateTablet for tablet 5afe19049971434284d13cd19109dc41 (DEFAULT_TABLE table=test-table [id=8385845da3584aaa90c4affd0260eb2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.561942 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5afe19049971434284d13cd19109dc41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.562861 3513 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Bootstrap starting.
I20260926 07:01:47.563508 3513 tablet_bootstrap.cc:654] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.563825 3513 log.cc:824] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:47.564162 3514 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Bootstrap starting.
I20260926 07:01:47.564515 3513 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: No bootstrap required, opened a new log
I20260926 07:01:47.564559 3513 ts_tablet_manager.cc:1397] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:47.564796 3514 tablet_bootstrap.cc:654] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.564926 3516 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Bootstrap starting.
I20260926 07:01:47.565130 3514 log.cc:824] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:47.565747 3516 tablet_bootstrap.cc:654] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.566069 3516 log.cc:824] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:47.566802 3516 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: No bootstrap required, opened a new log
I20260926 07:01:47.566740 3513 raft_consensus.cc:348] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.566869 3516 ts_tablet_manager.cc:1397] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:47.566910 3513 raft_consensus.cc:374] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.566958 3513 raft_consensus.cc:729] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b331f9f7fc9b4eefa688da729bfa4e36, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.567077 3513 consensus_queue.cc:260] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.567341 3513 ts_tablet_manager.cc:1428] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:47.567404 3351 heartbeater.cc:503] Master 127.1.217.126:33745 was elected leader, sending a full tablet report...
I20260926 07:01:47.568005 3514 tablet_bootstrap.cc:492] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: No bootstrap required, opened a new log
I20260926 07:01:47.568074 3514 ts_tablet_manager.cc:1397] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:01:47.568960 3516 raft_consensus.cc:348] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.569118 3516 raft_consensus.cc:374] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.569152 3516 raft_consensus.cc:729] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fe6d2eb29a44558dd09526db6d18c6, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.569263 3516 consensus_queue.cc:260] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.569515 3516 ts_tablet_manager.cc:1428] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:47.569969 3514 raft_consensus.cc:348] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.570083 3514 raft_consensus.cc:374] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.570103 3514 raft_consensus.cc:729] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e688ca81a84f7c9ded1d25484966c1, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.570211 3514 consensus_queue.cc:260] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.570432 3514 ts_tablet_manager.cc:1428] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:01:47.570528 3211 heartbeater.cc:503] Master 127.1.217.126:33745 was elected leader, sending a full tablet report...
I20260926 07:01:47.570559 3491 heartbeater.cc:503] Master 127.1.217.126:33745 was elected leader, sending a full tablet report...
W20260926 07:01:47.622836 3352 tablet.cc:2369] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:01:47.715926 3212 tablet.cc:2369] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:01:47.768421 3492 tablet.cc:2369] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:47.790906 3519 raft_consensus.cc:482] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:47.791030 3519 raft_consensus.cc:504] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.791370 3519 leader_election.cc:288] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:47.794380 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.791629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:37541 (local address 127.1.217.67:43101)
0926 07:01:47.791675 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.791677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.791714 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:01:47.791742 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.791743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.791754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.791855 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.792143 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.792642 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.793640 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.794021 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.794105 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.794114 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.794272 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.794310 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:47.794404 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.791557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:43101 (local address 127.1.217.66:37541)
0926 07:01:47.791611 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.791623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.791671 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.791866 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.791867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.791870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.791949 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.791951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.792739 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.792741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.793546 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.793548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.794114 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.794114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.794196 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.794200 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.794265 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.794336 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:01:47.794546 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.791658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:53541 (local address 127.1.217.65:43045)
0926 07:01:47.791698 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.791700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.791952 (+ 252us) server_negotiation.cc:407] Connection header received
0926 07:01:47.792007 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.792007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.792016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.792097 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.792265 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.792829 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.793823 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.794184 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.794268 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.794275 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.794447 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.794481 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:47.794564 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.791557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43045 (local address 127.1.217.66:53541)
0926 07:01:47.791884 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.791894 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.791949 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.792108 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.792108 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.792109 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.792165 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.792166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.792991 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.792993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.793727 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.793729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.794267 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.794268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.794370 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.794373 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.794442 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.794480 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":59,"thread_start_us":113,"threads_started":1}
I20260926 07:01:47.794736 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afe19049971434284d13cd19109dc41" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" is_pre_election: true
I20260926 07:01:47.794901 3445 raft_consensus.cc:2454] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 0.
I20260926 07:01:47.794862 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afe19049971434284d13cd19109dc41" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e688ca81a84f7c9ded1d25484966c1" is_pre_election: true
I20260926 07:01:47.794991 3165 raft_consensus.cc:2454] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 0.
I20260926 07:01:47.795149 3238 leader_election.cc:302] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b331f9f7fc9b4eefa688da729bfa4e36, d4fe6d2eb29a44558dd09526db6d18c6; no voters:
I20260926 07:01:47.795317 3519 raft_consensus.cc:2781] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:47.795404 3519 raft_consensus.cc:482] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:47.795436 3519 raft_consensus.cc:3037] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.796386 3519 raft_consensus.cc:504] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.796540 3519 leader_election.cc:288] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 election: Requested vote from peers 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:47.796712 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afe19049971434284d13cd19109dc41" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
I20260926 07:01:47.796723 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5afe19049971434284d13cd19109dc41" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e688ca81a84f7c9ded1d25484966c1"
I20260926 07:01:47.796830 3445 raft_consensus.cc:3037] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.796841 3165 raft_consensus.cc:3037] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.797511 3445 raft_consensus.cc:2454] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 1.
I20260926 07:01:47.797534 3165 raft_consensus.cc:2454] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 1.
I20260926 07:01:47.797674 3238 leader_election.cc:302] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b331f9f7fc9b4eefa688da729bfa4e36, d4fe6d2eb29a44558dd09526db6d18c6; no voters:
I20260926 07:01:47.797752 3519 raft_consensus.cc:2781] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:47.797917 3519 raft_consensus.cc:686] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 LEADER]: Becoming Leader. State: Replica: b331f9f7fc9b4eefa688da729bfa4e36, State: Running, Role: LEADER
I20260926 07:01:47.798020 3519 consensus_queue.cc:237] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.798882 3007 catalog_manager.cc:5495] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 reported cstate change: term changed from 0 to 1, leader changed from <none> to b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:47.844658 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.842083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.1.217.66:46751)
0926 07:01:47.842149 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.842151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.842160 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:47.842224 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.842225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.842248 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.842355 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:47.842518 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.842996 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.843797 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.843991 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.844067 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.844299 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:47.844390 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:47.844480 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.844521 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.844544 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:01:47.846560 3165 raft_consensus.cc:1261] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Refusing update from remote peer b331f9f7fc9b4eefa688da729bfa4e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:47.846560 3445 raft_consensus.cc:1261] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Refusing update from remote peer b331f9f7fc9b4eefa688da729bfa4e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:47.846903 3523 consensus_queue.cc:1035] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:47.847038 3519 consensus_queue.cc:1035] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:47.849261 3526 mvcc.cc:205] Tried to move back new op lower bound from 7333503417736704000 to 7333503417541476352. Current Snapshot: MvccSnapshot[applied={T|T < 7333503417736704000}]
I20260926 07:01:47.849812 3528 mvcc.cc:205] Tried to move back new op lower bound from 7333503417736704000 to 7333503417541476352. Current Snapshot: MvccSnapshot[applied={T|T < 7333503417736704000}]
I20260926 07:01:47.850320 3527 mvcc.cc:205] Tried to move back new op lower bound from 7333503417736704000 to 7333503417541476352. Current Snapshot: MvccSnapshot[applied={T|T < 7333503417736704000 or (T in {7333503417736704000})}]
I20260926 07:01:47.853138 3007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:47.853229 3007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:47.854023 3007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Sending DeleteTablet for 3 replicas of tablet 5afe19049971434284d13cd19109dc41
I20260926 07:01:47.854393 3419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afe19049971434284d13cd19109dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:47 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36760
I20260926 07:01:47.854440 3145 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afe19049971434284d13cd19109dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:47 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:37038
I20260926 07:01:47.854552 3007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:01:47.854628 3285 tablet_service.cc:1514] Processing DeleteTablet for tablet 5afe19049971434284d13cd19109dc41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:47 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:35830
W20260926 07:01:47.854707 3007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:47.854933 3536 tablet_replica.cc:330] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: stopping tablet replica
I20260926 07:01:47.855043 3536 raft_consensus.cc:2229] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:47.855161 3536 raft_consensus.cc:2258] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:47.855611 3537 tablet_replica.cc:330] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: stopping tablet replica
I20260926 07:01:47.855643 3536 ts_tablet_manager.cc:1905] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:47.855717 3537 raft_consensus.cc:2229] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:47.855813 3537 raft_consensus.cc:2258] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:47.855953 3538 tablet_replica.cc:330] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: stopping tablet replica
I20260926 07:01:47.856043 3538 raft_consensus.cc:2229] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:47.856158 3538 raft_consensus.cc:2258] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:47.856181 3145 tablet_service.cc:1467] Processing CreateTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (DEFAULT_TABLE table=test-table [id=bf853df94f4b441a9e86524f0419b80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.856289 3145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.856567 3538 ts_tablet_manager.cc:1905] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:47.857143 3537 ts_tablet_manager.cc:1905] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:47.857666 3536 ts_tablet_manager.cc:1918] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:47.857733 3536 log.cc:1196] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/5afe19049971434284d13cd19109dc41
I20260926 07:01:47.857728 3285 tablet_service.cc:1467] Processing CreateTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (DEFAULT_TABLE table=test-table [id=bf853df94f4b441a9e86524f0419b80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.857863 3285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.857996 3536 ts_tablet_manager.cc:1939] T 5afe19049971434284d13cd19109dc41 P d4fe6d2eb29a44558dd09526db6d18c6: Deleting consensus metadata
I20260926 07:01:47.858517 2994 catalog_manager.cc:4841] TS d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101): tablet 5afe19049971434284d13cd19109dc41 (table test-table [id=8385845da3584aaa90c4affd0260eb2f]) successfully deleted
I20260926 07:01:47.858578 3538 ts_tablet_manager.cc:1918] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:47.858634 3538 log.cc:1196] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/5afe19049971434284d13cd19109dc41
I20260926 07:01:47.858686 3537 ts_tablet_manager.cc:1918] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:01:47.858744 3537 log.cc:1196] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/5afe19049971434284d13cd19109dc41
I20260926 07:01:47.858922 3537 ts_tablet_manager.cc:1939] T 5afe19049971434284d13cd19109dc41 P 42e688ca81a84f7c9ded1d25484966c1: Deleting consensus metadata
I20260926 07:01:47.859139 3513 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Bootstrap starting.
I20260926 07:01:47.859174 3419 tablet_service.cc:1467] Processing CreateTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (DEFAULT_TABLE table=test-table [id=bf853df94f4b441a9e86524f0419b80e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:47.859195 3538 ts_tablet_manager.cc:1939] T 5afe19049971434284d13cd19109dc41 P b331f9f7fc9b4eefa688da729bfa4e36: Deleting consensus metadata
I20260926 07:01:47.859267 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.859543 3513 tablet_bootstrap.cc:654] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.860167 3513 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: No bootstrap required, opened a new log
I20260926 07:01:47.860209 3513 ts_tablet_manager.cc:1397] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:47.860347 3516 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Bootstrap starting.
I20260926 07:01:47.860380 3514 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Bootstrap starting.
I20260926 07:01:47.860365 3513 raft_consensus.cc:348] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.860430 3513 raft_consensus.cc:374] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.860452 3513 raft_consensus.cc:729] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b331f9f7fc9b4eefa688da729bfa4e36, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.860495 3513 consensus_queue.cc:260] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.860579 3513 ts_tablet_manager.cc:1428] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:47.860787 3514 tablet_bootstrap.cc:654] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.860800 3516 tablet_bootstrap.cc:654] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.861210 2995 catalog_manager.cc:4841] TS b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66:46751): tablet 5afe19049971434284d13cd19109dc41 (table test-table [id=8385845da3584aaa90c4affd0260eb2f]) successfully deleted
I20260926 07:01:47.861296 2995 catalog_manager.cc:4841] TS 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045): tablet 5afe19049971434284d13cd19109dc41 (table test-table [id=8385845da3584aaa90c4affd0260eb2f]) successfully deleted
I20260926 07:01:47.861876 3514 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: No bootstrap required, opened a new log
I20260926 07:01:47.861928 3514 ts_tablet_manager.cc:1397] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:47.862057 3516 tablet_bootstrap.cc:492] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: No bootstrap required, opened a new log
I20260926 07:01:47.862111 3516 ts_tablet_manager.cc:1397] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:47.862069 3514 raft_consensus.cc:348] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.862179 3514 raft_consensus.cc:374] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.862206 3514 raft_consensus.cc:729] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e688ca81a84f7c9ded1d25484966c1, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.862265 3514 consensus_queue.cc:260] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.862254 3516 raft_consensus.cc:348] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.862321 3516 raft_consensus.cc:374] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.862363 3516 raft_consensus.cc:729] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fe6d2eb29a44558dd09526db6d18c6, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.862356 3514 ts_tablet_manager.cc:1428] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:47.862404 3516 consensus_queue.cc:260] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.862500 3516 ts_tablet_manager.cc:1428] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:47.929848 3521 raft_consensus.cc:482] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:47.929962 3521 raft_consensus.cc:504] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.930315 3521 leader_election.cc:288] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66:46751), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:47.934052 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.930705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:53487 (local address 127.1.217.67:43101)
0926 07:01:47.930747 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.930749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.930758 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:47.930834 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.930836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.930848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.930880 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.931130 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.931871 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.933090 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.933625 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.933702 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.933711 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.933934 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.933979 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:47.934060 3542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.930503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:43101 (local address 127.1.217.65:53487)
0926 07:01:47.930675 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.930689 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.930745 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.930959 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.930959 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.930960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.931037 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.931039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.931981 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.931982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.932986 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.932987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.933767 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.933769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.933848 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.933852 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.933915 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.933955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260926 07:01:47.934345 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d4c6adc2c548e99e7a1ebb75a4c1ae" candidate_uuid: "42e688ca81a84f7c9ded1d25484966c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" is_pre_election: true
I20260926 07:01:47.934450 3445 raft_consensus.cc:2454] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e688ca81a84f7c9ded1d25484966c1 in term 0.
I20260926 07:01:47.934530 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.930585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:48007 (local address 127.1.217.66:46751)
0926 07:01:47.930632 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.930634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.930802 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:01:47.930859 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.930860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.930871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.930937 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:47.931989 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.932674 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.933663 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.934124 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.934202 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.934211 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.934418 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.934458 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:47.934643 3098 leader_election.cc:302] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e688ca81a84f7c9ded1d25484966c1, d4fe6d2eb29a44558dd09526db6d18c6; no voters:
I20260926 07:01:47.934757 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.930503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46751 (local address 127.1.217.65:48007)
0926 07:01:47.930693 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.930755 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.930802 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.930938 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.930939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.930941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:47.931037 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.931040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.932800 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.932801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.933568 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.933570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.934249 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.934250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.934330 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.934334 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.934598 (+ 264us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.934640 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":20,"thread_start_us":96,"threads_started":1}
I20260926 07:01:47.934801 3521 raft_consensus.cc:2781] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:47.934849 3521 raft_consensus.cc:482] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:47.934876 3521 raft_consensus.cc:3037] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.935539 3521 raft_consensus.cc:504] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.935674 3521 leader_election.cc:288] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [CANDIDATE]: Term 1 election: Requested vote from peers b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66:46751), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:47.935860 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d4c6adc2c548e99e7a1ebb75a4c1ae" candidate_uuid: "42e688ca81a84f7c9ded1d25484966c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
I20260926 07:01:47.935941 3445 raft_consensus.cc:3037] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:47.935989 3304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d4c6adc2c548e99e7a1ebb75a4c1ae" candidate_uuid: "42e688ca81a84f7c9ded1d25484966c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b331f9f7fc9b4eefa688da729bfa4e36"
I20260926 07:01:47.935969 3305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d4c6adc2c548e99e7a1ebb75a4c1ae" candidate_uuid: "42e688ca81a84f7c9ded1d25484966c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" is_pre_election: true
I20260926 07:01:47.936111 3305 raft_consensus.cc:2454] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e688ca81a84f7c9ded1d25484966c1 in term 0.
I20260926 07:01:47.936728 3445 raft_consensus.cc:2454] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e688ca81a84f7c9ded1d25484966c1 in term 1.
I20260926 07:01:47.936952 3098 leader_election.cc:302] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42e688ca81a84f7c9ded1d25484966c1, d4fe6d2eb29a44558dd09526db6d18c6; no voters: b331f9f7fc9b4eefa688da729bfa4e36
I20260926 07:01:47.937112 3521 raft_consensus.cc:2781] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:47.937275 3521 raft_consensus.cc:686] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 1 LEADER]: Becoming Leader. State: Replica: 42e688ca81a84f7c9ded1d25484966c1, State: Running, Role: LEADER
I20260926 07:01:47.937357 3521 consensus_queue.cc:237] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.938140 3008 catalog_manager.cc:5495] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "42e688ca81a84f7c9ded1d25484966c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } health_report { overall_health: UNKNOWN } } }
Sep 26 07:01:47 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406106, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:47.988891 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.981480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33745 (local address 127.0.0.1:49964)
0926 07:01:47.981638 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:47.981649 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:47.981717 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:47.981859 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:47.981859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:47.982151 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:47.982198 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.982200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.982822 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.982823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.983352 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:47.983354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.983682 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.983683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:47.983730 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.984209 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:47.984213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:47.985631 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:47.987517 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.987537 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.987540 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.987855 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.988075 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:47.988076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:47.988076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:47.988163 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:47.988483 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:47.988490 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:47.988545 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 07:01:47.988627 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 07:01:47.988742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260926 07:01:47.988976 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:47.981545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.1.217.126:33745)
0926 07:01:47.981606 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:47.981609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:47.981716 (+ 107us) server_negotiation.cc:407] Connection header received
0926 07:01:47.981787 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:47.981788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:47.981798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:47.981841 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:47.982267 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.982748 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.983436 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:47.983612 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:47.983669 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:47.985716 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:47.985717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:47.985718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:47.985724 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:47.987429 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.987941 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.987942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.987943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.988005 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:47.988253 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:47.988254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:47.988254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:47.988419 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:47.988491 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:47.988741 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 07:01:47.988856 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:01:47.991382 3008 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:49964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:01:47.991622 3008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:47.993050 3419 tablet_service.cc:1467] Processing CreateTablet for tablet 5238377cefe84d77ae12fd1e272bf0fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4883e68d0e3744eda02fad23d246568b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:47.993117 3145 tablet_service.cc:1467] Processing CreateTablet for tablet 5238377cefe84d77ae12fd1e272bf0fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4883e68d0e3744eda02fad23d246568b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:47.993214 3419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5238377cefe84d77ae12fd1e272bf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.993227 3145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5238377cefe84d77ae12fd1e272bf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.994701 3514 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: Bootstrap starting.
I20260926 07:01:47.994701 3516 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: Bootstrap starting.
I20260926 07:01:47.994797 3285 tablet_service.cc:1467] Processing CreateTablet for tablet 5238377cefe84d77ae12fd1e272bf0fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4883e68d0e3744eda02fad23d246568b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:47.994926 3285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5238377cefe84d77ae12fd1e272bf0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:47.995141 3516 tablet_bootstrap.cc:654] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.995141 3514 tablet_bootstrap.cc:654] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.995831 3516 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: No bootstrap required, opened a new log
I20260926 07:01:47.995904 3516 ts_tablet_manager.cc:1397] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:47.996027 3514 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: No bootstrap required, opened a new log
I20260926 07:01:47.996075 3514 ts_tablet_manager.cc:1397] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:47.996094 3516 raft_consensus.cc:348] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.996178 3516 raft_consensus.cc:374] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.996203 3516 raft_consensus.cc:729] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4fe6d2eb29a44558dd09526db6d18c6, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.996212 3514 raft_consensus.cc:348] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.996259 3513 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: Bootstrap starting.
I20260926 07:01:47.996250 3516 consensus_queue.cc:260] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.996280 3514 raft_consensus.cc:374] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.996304 3514 raft_consensus.cc:729] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e688ca81a84f7c9ded1d25484966c1, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.996356 3516 ts_tablet_manager.cc:1428] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:47.996346 3514 consensus_queue.cc:260] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.996435 3514 ts_tablet_manager.cc:1428] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.996491 3520 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:47.996577 3520 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:47.996599 3513 tablet_bootstrap.cc:654] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:47.996754 3521 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:47.996886 3521 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:47.997611 3513 tablet_bootstrap.cc:492] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: No bootstrap required, opened a new log
I20260926 07:01:47.997669 3513 ts_tablet_manager.cc:1397] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:47.997784 3513 raft_consensus.cc:348] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.997852 3513 raft_consensus.cc:374] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:47.997866 3513 raft_consensus.cc:729] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b331f9f7fc9b4eefa688da729bfa4e36, State: Initialized, Role: FOLLOWER
I20260926 07:01:47.997922 3513 consensus_queue.cc:260] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:47.998014 3513 ts_tablet_manager.cc:1428] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:47.998059 3531 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:47.998154 3531 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:48.029193 3531 raft_consensus.cc:482] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:48.029292 3531 raft_consensus.cc:504] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:48.029441 3531 leader_election.cc:288] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:48.029654 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5238377cefe84d77ae12fd1e272bf0fa" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e688ca81a84f7c9ded1d25484966c1" is_pre_election: true
I20260926 07:01:48.029680 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5238377cefe84d77ae12fd1e272bf0fa" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" is_pre_election: true
I20260926 07:01:48.029774 3445 raft_consensus.cc:2454] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 0.
I20260926 07:01:48.029778 3165 raft_consensus.cc:2454] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 0.
I20260926 07:01:48.029950 3238 leader_election.cc:302] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b331f9f7fc9b4eefa688da729bfa4e36, d4fe6d2eb29a44558dd09526db6d18c6; no voters:
I20260926 07:01:48.030072 3531 raft_consensus.cc:2781] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:48.030115 3531 raft_consensus.cc:482] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:48.030129 3531 raft_consensus.cc:3037] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:48.030889 3531 raft_consensus.cc:504] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:48.031070 3531 leader_election.cc:288] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 election: Requested vote from peers 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045), d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101)
I20260926 07:01:48.031211 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5238377cefe84d77ae12fd1e272bf0fa" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4fe6d2eb29a44558dd09526db6d18c6"
I20260926 07:01:48.031283 3445 raft_consensus.cc:3037] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:48.031239 3165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5238377cefe84d77ae12fd1e272bf0fa" candidate_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e688ca81a84f7c9ded1d25484966c1"
I20260926 07:01:48.031343 3165 raft_consensus.cc:3037] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:48.031880 3445 raft_consensus.cc:2454] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 1.
I20260926 07:01:48.032009 3165 raft_consensus.cc:2454] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b331f9f7fc9b4eefa688da729bfa4e36 in term 1.
I20260926 07:01:48.032037 3238 leader_election.cc:302] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b331f9f7fc9b4eefa688da729bfa4e36, d4fe6d2eb29a44558dd09526db6d18c6; no voters:
I20260926 07:01:48.032123 3531 raft_consensus.cc:2781] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:48.032171 3531 raft_consensus.cc:686] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 LEADER]: Becoming Leader. State: Replica: b331f9f7fc9b4eefa688da729bfa4e36, State: Running, Role: LEADER
I20260926 07:01:48.032224 3531 consensus_queue.cc:237] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } }
I20260926 07:01:48.032481 3519 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: TxnStatusTablet state changed. Reason: New leader b331f9f7fc9b4eefa688da729bfa4e36. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.032588 3519 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:48.032727 3556 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:01:48.032879 3556 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:01:48.032794 3007 catalog_manager.cc:5495] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 reported cstate change: term changed from 0 to 1, leader changed from <none> to b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:48.044752 3551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.042324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46751 (local address 127.0.0.1:35848)
0926 07:01:48.042402 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:48.042413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:48.042465 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:48.042641 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:48.042642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:48.042643 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:48.042694 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.042696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.043312 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.043314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.043893 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.043895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.044278 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.044279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.044345 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.044363 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.044562 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.044571 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:01:48.044626 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:01:48.044672 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 07:01:48.044729 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.042382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.1.217.66:46751)
0926 07:01:48.042439 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.042441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.042451 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:48.042543 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.042544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.042555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.042633 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:48.042754 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.043221 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.043977 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.044168 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.044280 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.044458 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.044500 (+ 42us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.044562 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.044631 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.044652 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:01:48.045797 3165 raft_consensus.cc:1261] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Refusing update from remote peer b331f9f7fc9b4eefa688da729bfa4e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:48.045976 3445 raft_consensus.cc:1261] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Refusing update from remote peer b331f9f7fc9b4eefa688da729bfa4e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:48.046020 3519 consensus_queue.cc:1035] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:48.046167 3519 consensus_queue.cc:1035] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:48.046646 3521 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: TxnStatusTablet state changed. Reason: New leader b331f9f7fc9b4eefa688da729bfa4e36. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.046767 3521 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:48.047168 3520 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: TxnStatusTablet state changed. Reason: New leader b331f9f7fc9b4eefa688da729bfa4e36. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.047269 3520 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:48.047297 3519 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.047370 3519 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:48.047984 3531 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.048059 3531 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P b331f9f7fc9b4eefa688da729bfa4e36: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:48.048365 3521 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.048446 3521 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P 42e688ca81a84f7c9ded1d25484966c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:48.048470 3520 tablet_replica.cc:439] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e688ca81a84f7c9ded1d25484966c1" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43045 } } peers { permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 } } peers { permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 } } }
I20260926 07:01:48.048554 3520 tablet_replica.cc:442] T 5238377cefe84d77ae12fd1e272bf0fa P d4fe6d2eb29a44558dd09526db6d18c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:48.052372 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.049571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37054 (local address 127.1.217.65:43045)
0926 07:01:48.049613 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.049615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.049624 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:48.049705 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.049706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.049725 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.049813 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:48.049946 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.050565 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.051655 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.051854 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.051912 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.052077 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.052182 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.052235 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.052306 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.052323 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:01:48.057320 3512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.054774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:54461 (local address 127.1.217.66:46751)
0926 07:01:48.054816 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.054818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.054914 (+ 96us) server_negotiation.cc:407] Connection header received
0926 07:01:48.054956 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.054956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.054965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.055006 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:48.055141 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.055575 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.056674 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.056919 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.056972 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.057148 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.057171 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.057208 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.057254 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.057271 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:48.057389 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.054676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46751 (local address 127.1.217.65:54461)
0926 07:01:48.054833 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:48.054844 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:48.054903 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:48.055017 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:48.055017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:48.055019 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:48.055083 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.055084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.055646 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.055646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.056592 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.056594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.056984 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.056985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.057052 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.057066 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.057209 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.057211 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:48.057253 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 07:01:48.057295 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260926 07:01:48.059659 3304 raft_consensus.cc:3037] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:48.059679 3445 raft_consensus.cc:1261] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Refusing update from remote peer 42e688ca81a84f7c9ded1d25484966c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:48.059911 3544 consensus_queue.cc:1035] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4fe6d2eb29a44558dd09526db6d18c6" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:48.060535 3304 raft_consensus.cc:1261] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Refusing update from remote peer 42e688ca81a84f7c9ded1d25484966c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:48.060801 3544 consensus_queue.cc:1035] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b331f9f7fc9b4eefa688da729bfa4e36" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:48.068922 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.066072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:43311 (local address 127.1.217.65:43045)
0926 07:01:48.066112 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.066114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.066316 (+ 202us) server_negotiation.cc:407] Connection header received
0926 07:01:48.066374 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.066375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.066386 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.066447 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:48.066548 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.067051 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.068248 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.068428 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.068494 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.068670 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.068698 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.068737 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.068844 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.068863 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:48.069000 3566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.066026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43045 (local address 127.1.217.66:43311)
0926 07:01:48.066242 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:48.066255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:48.066322 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:48.066443 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:48.066444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:48.066445 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:48.066492 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.066494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.067120 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.067121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.068126 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:48.068129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.068508 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.068509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:48.068577 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.068592 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:48.068756 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:48.068758 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:48.068841 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:01:48.068897 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":91,"threads_started":1}
I20260926 07:01:48.078408 3006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:48.078522 3006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:48.079232 3006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d258f74226db4bc9b6adf37fe5bd93dd: Sending DeleteTablet for 3 replicas of tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae
I20260926 07:01:48.079558 3145 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:48 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:37038
I20260926 07:01:48.079654 3537 tablet_replica.cc:330] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: stopping tablet replica
I20260926 07:01:48.079704 3537 raft_consensus.cc:2229] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:48.079775 3537 raft_consensus.cc:2258] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:48.080096 3537 ts_tablet_manager.cc:1905] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:48.081003 3285 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:48 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:35830
I20260926 07:01:48.081110 3538 tablet_replica.cc:330] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: stopping tablet replica
I20260926 07:01:48.081184 3538 raft_consensus.cc:2229] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:48.081228 3538 raft_consensus.cc:2258] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:48.081494 3538 ts_tablet_manager.cc:1905] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:48.081811 3537 ts_tablet_manager.cc:1918] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:48.081862 3537 log.cc:1196] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/45d4c6adc2c548e99e7a1ebb75a4c1ae
I20260926 07:01:48.082068 3537 ts_tablet_manager.cc:1939] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P 42e688ca81a84f7c9ded1d25484966c1: Deleting consensus metadata
I20260926 07:01:48.082194 3419 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:48 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36760
I20260926 07:01:48.082306 3536 tablet_replica.cc:330] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: stopping tablet replica
I20260926 07:01:48.082347 3536 raft_consensus.cc:2229] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:48.082383 3536 raft_consensus.cc:2258] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:48.082950 3538 ts_tablet_manager.cc:1918] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:48.082998 3538 log.cc:1196] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/45d4c6adc2c548e99e7a1ebb75a4c1ae
I20260926 07:01:48.083197 3538 ts_tablet_manager.cc:1939] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P b331f9f7fc9b4eefa688da729bfa4e36: Deleting consensus metadata
I20260926 07:01:48.083299 2995 catalog_manager.cc:4841] TS 42e688ca81a84f7c9ded1d25484966c1 (127.1.217.65:43045): tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (table test-table [id=bf853df94f4b441a9e86524f0419b80e]) successfully deleted
I20260926 07:01:48.083563 3536 ts_tablet_manager.cc:1905] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:48.083671 2995 catalog_manager.cc:4841] TS b331f9f7fc9b4eefa688da729bfa4e36 (127.1.217.66:46751): tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (table test-table [id=bf853df94f4b441a9e86524f0419b80e]) successfully deleted
I20260926 07:01:48.085115 3536 ts_tablet_manager.cc:1918] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:48.085176 3536 log.cc:1196] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/45d4c6adc2c548e99e7a1ebb75a4c1ae
I20260926 07:01:48.085391 3536 ts_tablet_manager.cc:1939] T 45d4c6adc2c548e99e7a1ebb75a4c1ae P d4fe6d2eb29a44558dd09526db6d18c6: Deleting consensus metadata
I20260926 07:01:48.085700 2994 catalog_manager.cc:4841] TS d4fe6d2eb29a44558dd09526db6d18c6 (127.1.217.67:43101): tablet 45d4c6adc2c548e99e7a1ebb75a4c1ae (table test-table [id=bf853df94f4b441a9e86524f0419b80e]) successfully deleted
Sep 26 07:01:48 dist-test-slave-t1mc krb5kdc[2963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406107, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:01:48.096118 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.085854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.1.217.65:43045)
0926 07:01:48.085897 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.085899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.086100 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:01:48.086147 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.086148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.086158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.086224 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:48.086899 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.087643 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.088695 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.089009 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.089083 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.092731 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:48.092738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:48.092739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:48.092747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:48.094783 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:48.095197 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:48.095201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:48.095202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:48.095282 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:48.095506 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:48.095507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:48.095507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:48.095679 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:48.095823 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.095908 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.095993 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 07:01:48.096473 3201 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37060
I20260926 07:01:48.108565 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:48.101063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.1.217.126:33745)
0926 07:01:48.101132 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:48.101134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:48.101202 (+ 68us) server_negotiation.cc:407] Connection header received
0926 07:01:48.101266 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:48.101267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:48.101278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:48.101330 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:48.101976 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.102749 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.103792 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:48.104086 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:48.104159 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:48.105601 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:48.105603 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:48.105603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:48.105610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:48.107331 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:48.107716 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:48.107717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:48.107718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:48.107788 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:48.108009 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:48.108010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:48.108010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:48.108143 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:48.108238 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:48.108395 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:01:48.108477 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 07:01:48.108952 3006 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49966
I20260926 07:01:48.109678 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3080
I20260926 07:01:48.116189 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3220
I20260926 07:01:48.122397 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3360
I20260926 07:01:48.126921 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 2976
2026-09-26T07:01:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2416 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:48 dist-test-slave-t1mc krb5kdc[3576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:48 dist-test-slave-t1mc krb5kdc[3576](info): set up 2 sockets
Sep 26 07:01:48 dist-test-slave-t1mc krb5kdc[3576](info): commencing operation
krb5kdc: starting...
W20260926 07:01:49.152359 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:49.269683 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37641
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:37641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:49.349305 3589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:49.349506 3589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:49.349527 3589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:49.350615 3589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:49.350652 3589 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:01:49.350664 3589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:49.350677 3589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:49.350689 3589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:49.350737 3589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:49.352137 3589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:37641
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:49.352383 3589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:49.352622 3589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:49.357477 3597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.357484 3594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.357491 3595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:49.357800 3589 server_base.cc:1034] running on GCE node
I20260926 07:01:49.358021 3589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:49.358343 3589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:49.359511 3589 hybrid_clock.cc:648] HybridClock initialized: now 1790406109359500 us; error 26 us; skew 500 ppm
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:49.362064 3589 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:49.362530 3589 webserver.cc:460] Webserver started at http://127.1.217.126:34531/ using document root <none> and password file <none>
I20260926 07:01:49.362747 3589 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:49.362815 3589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:49.362962 3589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:49.364045 3589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "5ad47ea6d9ee451f91fcdcc539673c95"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "f5050c502aae312a08e0ee423920a795"
server_key_iv: "3ba8c010c93f663781c3cc8fce6aa8c2"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.364403 3589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "5ad47ea6d9ee451f91fcdcc539673c95"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "f5050c502aae312a08e0ee423920a795"
server_key_iv: "3ba8c010c93f663781c3cc8fce6aa8c2"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.365790 3589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:49.366557 3604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.366724 3589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:49.366828 3589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "5ad47ea6d9ee451f91fcdcc539673c95"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "f5050c502aae312a08e0ee423920a795"
server_key_iv: "3ba8c010c93f663781c3cc8fce6aa8c2"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.366901 3589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:49.378108 3589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:49.380330 3589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:49.380486 3589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:49.385147 3589 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:37641
I20260926 07:01:49.385191 3666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:37641 every 8 connection(s)
I20260926 07:01:49.385519 3589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:49.386173 3667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:49.388268 3667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Bootstrap starting.
I20260926 07:01:49.388913 3667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:49.389140 3667 log.cc:824] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:49.389780 3667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: No bootstrap required, opened a new log
I20260926 07:01:49.391744 3667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } }
I20260926 07:01:49.391887 3667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:49.391909 3667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad47ea6d9ee451f91fcdcc539673c95, State: Initialized, Role: FOLLOWER
I20260926 07:01:49.392045 3667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } }
I20260926 07:01:49.392128 3667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:49.392151 3667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:49.392179 3667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:49.392642 3667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } }
I20260926 07:01:49.392724 3667 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad47ea6d9ee451f91fcdcc539673c95; no voters:
I20260926 07:01:49.392931 3667 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:49.392982 3670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:49.393181 3670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [term 1 LEADER]: Becoming Leader. State: Replica: 5ad47ea6d9ee451f91fcdcc539673c95, State: Running, Role: LEADER
I20260926 07:01:49.393194 3667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:49.393299 3670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } }
I20260926 07:01:49.393651 3671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } } }
I20260926 07:01:49.393637 3672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad47ea6d9ee451f91fcdcc539673c95. Latest consensus state: current_term: 1 leader_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad47ea6d9ee451f91fcdcc539673c95" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37641 } } }
I20260926 07:01:49.393749 3671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:49.393757 3672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:49.393965 3678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:49.394338 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3589
I20260926 07:01:49.394429 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:49.394513 3678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:49.394598 1893 external_mini_cluster.cc:1428] Setting key df2f267a00841b0022cac468130a8dbf
I20260926 07:01:49.396281 3678 catalog_manager.cc:1347] Generated new cluster ID: 0db4dfac7c0a4764a13830aa13dac39a
I20260926 07:01:49.396332 3678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:49.407655 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.394996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.1.217.126:37641)
0926 07:01:49.396908 (+ 1912us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.396918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.396980 (+ 62us) server_negotiation.cc:407] Connection header received
0926 07:01:49.397145 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.397153 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.397207 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.397352 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.398012 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.399456 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.400475 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.401185 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.401301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.404001 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.404005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.404005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.404017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.405923 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.406342 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.406345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.406345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.406445 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.406652 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.406652 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.406652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.406932 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.407068 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.407207 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.407435 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1863,"thread_start_us":76,"threads_started":1}
I20260926 07:01:49.428876 3678 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:49.429621 3678 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:49.433887 3678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Generated new TSK 0
I20260926 07:01:49.434069 3678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:49.467545 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37641
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:49.548087 3693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:49.548280 3693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:49.548303 3693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:49.549988 3693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:49.550029 3693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:49.550050 3693 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:01:49.550062 3693 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:49.550112 3693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:49.551497 3693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:49.551752 3693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:49.551993 3693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:49.557019 3699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:49.557097 3693 server_base.cc:1034] running on GCE node
W20260926 07:01:49.557036 3698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.557032 3701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:49.557405 3693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:49.557590 3693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:49.558754 3693 hybrid_clock.cc:648] HybridClock initialized: now 1790406109558744 us; error 26 us; skew 500 ppm
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:49.561111 3693 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:49.561479 3693 webserver.cc:460] Webserver started at http://127.1.217.65:42165/ using document root <none> and password file <none>
I20260926 07:01:49.561697 3693 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:49.561735 3693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:49.561817 3693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:49.562786 3693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "d229fd20700d9a372375cc8e3f22c113"
server_key_iv: "f10e0775c62fd1a046584ec3e4ca1c3c"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.563063 3693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "d229fd20700d9a372375cc8e3f22c113"
server_key_iv: "f10e0775c62fd1a046584ec3e4ca1c3c"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.564287 3693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:01:49.564977 3708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.565158 3693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:01:49.565266 3693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "d229fd20700d9a372375cc8e3f22c113"
server_key_iv: "f10e0775c62fd1a046584ec3e4ca1c3c"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.565339 3693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:49.574684 3693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:49.577059 3693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:49.577226 3693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:49.577725 3693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:49.577805 3693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.577857 3693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:49.577888 3693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.584162 3693 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:42813
I20260926 07:01:49.584254 3823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:42813 every 8 connection(s)
I20260926 07:01:49.584529 3693 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:49.587996 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.578261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:48865 (local address 127.1.217.126:37641)
0926 07:01:49.578307 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.578309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.578554 (+ 245us) server_negotiation.cc:407] Connection header received
0926 07:01:49.579087 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.579088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.579105 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.579177 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.580013 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.580871 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.582442 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.582666 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.582747 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.585345 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.585347 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.585347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.585357 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.586609 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.587009 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.587010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.587011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.587078 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.587343 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.587344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.587344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.587463 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.587538 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.587810 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.587905 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:49.588166 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.578250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.65:48865)
0926 07:01:49.578431 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.578467 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.578979 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.579243 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.579247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.579618 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.579923 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.579930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.580966 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.580966 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.582355 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.582358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.582751 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.582752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.582920 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.583490 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.583502 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.585246 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.586704 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.586710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.586712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.586932 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.587148 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.587149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.587149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.587264 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.587529 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.587536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.587679 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.587789 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.587940 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260926 07:01:49.588759 3824 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37641
I20260926 07:01:49.588909 3824 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:49.589087 3824 heartbeater.cc:511] Master 127.1.217.126:37641 requested a full tablet report, sending...
I20260926 07:01:49.589613 3620 ts_manager.cc:194] Registered new tserver with Master: cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813)
I20260926 07:01:49.590400 3620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:48865
I20260926 07:01:49.592592 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3693
I20260926 07:01:49.592665 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:49.592870 1893 external_mini_cluster.cc:1428] Setting key f803d70a5a27b01d095fe6a41508eb39
I20260926 07:01:49.595530 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.589770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.65:37009)
0926 07:01:49.589932 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.589944 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.589999 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.590131 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.590132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.590290 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.590337 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.590338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.591074 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.591075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.591592 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.591593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.591970 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.591971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.592046 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.592403 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.592407 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.592732 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.594502 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.594503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.594504 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.594754 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.594989 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.594989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.594989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.595044 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.595307 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.595308 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.595343 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.595393 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.595467 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260926 07:01:49.595641 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.589971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:37009 (local address 127.1.217.126:37641)
0926 07:01:49.590016 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.590018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.590025 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:49.590057 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.590057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.590066 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.590106 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.590420 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.590984 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.591685 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.591888 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.591970 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.592868 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.592869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.592870 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.592876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.594400 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.594841 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.594842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.594843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.594909 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.595105 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.595106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.595106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.595243 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.595314 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.595440 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.595527 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:49.617419 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37641
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:49.698141 3833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:49.698347 3833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:49.698372 3833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:49.699440 3833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:49.699471 3833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:49.699491 3833 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:01:49.699503 3833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:49.699519 3833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:49.700866 3833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:49.701033 3833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:49.701200 3833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:49.705914 3838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.705909 3839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.705909 3841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:49.706104 3833 server_base.cc:1034] running on GCE node
I20260926 07:01:49.706451 3833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:49.706743 3833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:49.707908 3833 hybrid_clock.cc:648] HybridClock initialized: now 1790406109707892 us; error 26 us; skew 500 ppm
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:49.710371 3833 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:49.710835 3833 webserver.cc:460] Webserver started at http://127.1.217.66:34441/ using document root <none> and password file <none>
I20260926 07:01:49.711053 3833 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:49.711118 3833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:49.711256 3833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:49.712271 3833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "0c9614e8e2c343c7a590b6406ff051f0"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "0e6ed1b555b31654eb1683f9090895e4"
server_key_iv: "fd9a2c2f4c313bdc1d97532176878eb8"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.712637 3833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9614e8e2c343c7a590b6406ff051f0"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "0e6ed1b555b31654eb1683f9090895e4"
server_key_iv: "fd9a2c2f4c313bdc1d97532176878eb8"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.713989 3833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:49.714715 3848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.714926 3833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:49.715019 3833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "0c9614e8e2c343c7a590b6406ff051f0"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "0e6ed1b555b31654eb1683f9090895e4"
server_key_iv: "fd9a2c2f4c313bdc1d97532176878eb8"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.715101 3833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:49.728514 3833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:49.730759 3833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:49.730917 3833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:49.731457 3833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:49.731515 3833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.731555 3833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:49.731580 3833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:49.738965 3833 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:34241
I20260926 07:01:49.739320 3833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:49.739284 3963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:34241 every 8 connection(s)
I20260926 07:01:49.741434 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.731859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:55793 (local address 127.1.217.126:37641)
0926 07:01:49.731912 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.731914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.732219 (+ 305us) server_negotiation.cc:407] Connection header received
0926 07:01:49.732950 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.732951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.732962 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.733017 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.733930 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.734422 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.736091 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.736303 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.736398 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.738679 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.738681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.738681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.738690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.740275 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.740630 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.740630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.740631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.740676 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.740911 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.740912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.740912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.741006 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.741054 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.741318 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.741375 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:01:49.741602 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.731856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.66:55793)
0926 07:01:49.732043 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.732095 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.732826 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.733125 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.733130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.733524 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.733830 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.733837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.734508 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.734509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.735989 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.735993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.736415 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.736417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.736556 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.737095 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.737106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.738583 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.740344 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.740348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.740349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.740566 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.740715 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.740716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.740716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.740851 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.741059 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.741063 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.741169 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.741261 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.741390 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260926 07:01:49.742050 3964 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37641
I20260926 07:01:49.742157 3964 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:49.742187 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3833
I20260926 07:01:49.742295 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:49.742359 3964 heartbeater.cc:511] Master 127.1.217.126:37641 requested a full tablet report, sending...
I20260926 07:01:49.742479 1893 external_mini_cluster.cc:1428] Setting key 2444fb9f7f993c7ec13ca9d32322bfce
I20260926 07:01:49.742766 3620 ts_manager.cc:194] Registered new tserver with Master: 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241)
I20260926 07:01:49.743281 3620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:55793
I20260926 07:01:49.750451 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.743063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:57191 (local address 127.1.217.126:37641)
0926 07:01:49.743098 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.743100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.744481 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 07:01:49.744507 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.744508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.744520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.744593 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.744908 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.745543 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.746687 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.746861 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.746918 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.747984 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.747985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.747985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.747991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.749343 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.749735 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.749736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.749736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.749782 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.749980 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.749980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.749980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.750100 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.750151 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.750338 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.750393 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:49.750573 3971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.743028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.66:57191)
0926 07:01:49.744387 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.744399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.744461 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.744595 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.744596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.744737 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.744844 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.744847 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.745661 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.745662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.746587 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.746589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.746940 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.746941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.747009 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.747440 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.747446 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.747890 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.749424 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.749425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.749426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.749655 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.749852 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.749853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.749854 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.749919 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.750165 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.750166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.750233 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.750323 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.750425 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1334,"thread_start_us":1308,"threads_started":1}
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:49.767935 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37641
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:49.848125 3973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:49.848321 3973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:49.848345 3973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:49.849414 3973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:49.849449 3973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:49.849469 3973 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 07:01:49.849481 3973 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:01:49.849498 3973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:49.850826 3973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:37641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:49.850993 3973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:49.851177 3973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:49.856230 3981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.856235 3978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:49.856235 3979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:49.856551 3973 server_base.cc:1034] running on GCE node
I20260926 07:01:49.856813 3973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:49.857081 3973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:49.858235 3973 hybrid_clock.cc:648] HybridClock initialized: now 1790406109858210 us; error 40 us; skew 500 ppm
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:49.860615 3973 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:49.861146 3973 webserver.cc:460] Webserver started at http://127.1.217.67:43519/ using document root <none> and password file <none>
I20260926 07:01:49.861392 3973 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:49.861460 3973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:49.861591 3973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:49.862614 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "c931b92d1c614b2fb52898369fa7455e"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "df4cc5e5b157d525e4259fef6aa0f6e2"
server_key_iv: "f7aaad92393fe2a524adb73d48279949"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.862974 3973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "c931b92d1c614b2fb52898369fa7455e"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "df4cc5e5b157d525e4259fef6aa0f6e2"
server_key_iv: "f7aaad92393fe2a524adb73d48279949"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.864279 3973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:49.865010 3988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.865168 3973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:49.865266 3973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "c931b92d1c614b2fb52898369fa7455e"
format_stamp: "Formatted at 2026-09-26 07:01:49 on dist-test-slave-t1mc"
server_key: "df4cc5e5b157d525e4259fef6aa0f6e2"
server_key_iv: "f7aaad92393fe2a524adb73d48279949"
server_key_version: "encryptionkey@0"
I20260926 07:01:49.865347 3973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:49.878779 3973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:49.880596 3973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:49.880760 3973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:49.881374 3973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:49.881433 3973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:49.881474 3973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:49.881495 3973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:49.888849 3973 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:39407
I20260926 07:01:49.888902 4103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:39407 every 8 connection(s)
I20260926 07:01:49.889305 3973 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:49.891309 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.881716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:39031 (local address 127.1.217.126:37641)
0926 07:01:49.881783 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.881786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.882040 (+ 254us) server_negotiation.cc:407] Connection header received
0926 07:01:49.882542 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.882543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.882557 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.882643 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.883309 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.883826 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.885290 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.885494 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.885578 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.888190 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.888192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.888192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.888200 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.889891 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.890363 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.890365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.890365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.890422 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.890676 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.890677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.890678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.890795 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.890865 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.891145 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.891222 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:01:49.891475 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.881733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.67:39031)
0926 07:01:49.881899 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.881939 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.882403 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.882694 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.882697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.882927 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.883202 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.883208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.883933 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.883934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.885205 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.885208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.885584 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.885585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.885717 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.886365 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.886372 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.888083 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.889982 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.889987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.889989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.890273 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.890492 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.890493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.890493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.890607 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.890869 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.890875 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.891004 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.891104 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.891252 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260926 07:01:49.891908 4104 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37641
I20260926 07:01:49.891988 4104 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:49.892168 4104 heartbeater.cc:511] Master 127.1.217.126:37641 requested a full tablet report, sending...
I20260926 07:01:49.892781 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 3973
I20260926 07:01:49.892879 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:49.892875 3621 ts_manager.cc:194] Registered new tserver with Master: c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:49.893080 1893 external_mini_cluster.cc:1428] Setting key f566efcf9b7dff0fce0fb5c5408adcc8
I20260926 07:01:49.893424 3621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:39031
I20260926 07:01:49.893954 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:01:49.899183 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.893293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:57953 (local address 127.1.217.126:37641)
0926 07:01:49.893334 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.893336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.893690 (+ 354us) server_negotiation.cc:407] Connection header received
0926 07:01:49.893821 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.893823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.893832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.893895 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.894289 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.894821 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.895574 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.895775 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.895857 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.896733 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.896735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.896735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.896742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.898057 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.898414 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.898415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.898416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.898471 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.898673 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.898674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.898674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.898786 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.898862 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.899017 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.899092 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:01:49.899199 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.892921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.1.217.67:57953)
0926 07:01:49.893611 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.893622 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.893717 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.893937 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.893938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.894134 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:49.894193 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.894195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.894918 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.894919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.895464 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.895466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.895859 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.895860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.895925 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.896375 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:49.896379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:49.896643 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:49.898157 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.898158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.898159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.898329 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.898543 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:49.898544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:49.898544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:49.898601 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:49.898854 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:49.898855 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:49.898899 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.898993 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.899094 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":71,"threads_started":1}
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:01:49 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:49.916527 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.909807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.1.217.126:37641)
0926 07:01:49.909865 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.909867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.909888 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:01:49.909920 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.909920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.909929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.909988 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:49.910522 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.911023 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.911723 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.911925 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.912012 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.914223 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:49.914224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:49.914224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:49.914232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:49.915402 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.915755 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.915757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.915757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.915814 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:49.916006 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:49.916007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:49.916007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:49.916136 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:49.916243 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.916345 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.916424 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:01:49.918284 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:49.919813 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:49.926287 4120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:39407 (local address 127.0.0.1:56264)
0926 07:01:49.922531 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.922540 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.922661 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.923183 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.923184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.923186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:49.923218 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.923219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.924373 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.924374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.925281 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.925283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.925975 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.925976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.926063 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926079 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.926153 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.926202 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":24,"thread_start_us":48,"threads_started":1}
I20260926 07:01:49.926293 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.1.217.67:39407)
0926 07:01:49.922996 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.922999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.923007 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:49.923067 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.923069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.923083 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.923166 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:49.923274 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.924295 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.925366 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.925879 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.925975 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926057 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.926171 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.926204 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":87,"threads_started":1}
I20260926 07:01:49.926578 4119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.0.0.1:36790)
0926 07:01:49.922487 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.922497 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.922661 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.922964 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.922964 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.922966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:49.923008 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.923009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.923979 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.923980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.925302 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.925304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.926331 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.926332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.926405 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926410 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.926474 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.926511 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260926 07:01:49.926653 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36790 (local address 127.1.217.65:42813)
0926 07:01:49.922545 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.922550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.922644 (+ 94us) server_negotiation.cc:407] Connection header received
0926 07:01:49.922787 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.922796 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.922813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.922897 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:49.923085 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.923885 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.925401 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.926172 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.926244 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.926505 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.926547 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260926 07:01:49.927137 4118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:34241 (local address 127.0.0.1:33266)
0926 07:01:49.922303 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:49.922345 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:49.922420 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:49.922899 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:49.922902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:49.922904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:49.922973 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.922975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.924269 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.924270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.925347 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:49.925348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.926896 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.926898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:49.926969 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926973 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:49.927036 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:01:49.927073 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260926 07:01:49.927053 4038 tablet_service.cc:1467] Processing CreateTablet for tablet f751ab63a55544e486c67f4aa97894ad (DEFAULT_TABLE table=test-table [id=711068f9a3cc4276a4620ea8099be229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:49.927204 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:49.922308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.1.217.66:34241)
0926 07:01:49.922724 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:49.922728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:49.922737 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:49.922777 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:49.922779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:49.922793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:49.922875 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:49.923063 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.924184 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.925426 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:49.926173 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:49.926244 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:49.926294 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:49.927071 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 07:01:49.927114 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":101,"threads_started":1}
I20260926 07:01:49.927417 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f751ab63a55544e486c67f4aa97894ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:49.927310 3739 tablet_service.cc:1467] Processing CreateTablet for tablet f751ab63a55544e486c67f4aa97894ad (DEFAULT_TABLE table=test-table [id=711068f9a3cc4276a4620ea8099be229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:49.927625 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f751ab63a55544e486c67f4aa97894ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:49.928056 3884 tablet_service.cc:1467] Processing CreateTablet for tablet f751ab63a55544e486c67f4aa97894ad (DEFAULT_TABLE table=test-table [id=711068f9a3cc4276a4620ea8099be229]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:49.928350 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f751ab63a55544e486c67f4aa97894ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:49.930141 4124 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Bootstrap starting.
I20260926 07:01:49.930896 4124 tablet_bootstrap.cc:654] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:49.931242 4125 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Bootstrap starting.
I20260926 07:01:49.931289 4124 log.cc:824] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:49.931901 4125 tablet_bootstrap.cc:654] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:49.931994 4127 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Bootstrap starting.
I20260926 07:01:49.932083 4124 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: No bootstrap required, opened a new log
I20260926 07:01:49.932144 4124 ts_tablet_manager.cc:1397] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:49.932377 4125 log.cc:824] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:49.932655 4127 tablet_bootstrap.cc:654] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:49.933010 4127 log.cc:824] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:49.933195 4125 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: No bootstrap required, opened a new log
I20260926 07:01:49.933271 4125 ts_tablet_manager.cc:1397] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:49.933745 4127 tablet_bootstrap.cc:492] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: No bootstrap required, opened a new log
I20260926 07:01:49.933826 4127 ts_tablet_manager.cc:1397] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:01:49.934275 4124 raft_consensus.cc:348] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.934445 4124 raft_consensus.cc:374] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:49.934478 4124 raft_consensus.cc:729] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Initialized, Role: FOLLOWER
I20260926 07:01:49.934604 4124 consensus_queue.cc:260] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.934887 4124 ts_tablet_manager.cc:1428] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:49.934916 4104 heartbeater.cc:503] Master 127.1.217.126:37641 was elected leader, sending a full tablet report...
I20260926 07:01:49.935349 4125 raft_consensus.cc:348] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.935500 4125 raft_consensus.cc:374] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:49.935532 4125 raft_consensus.cc:729] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Initialized, Role: FOLLOWER
I20260926 07:01:49.935649 4125 consensus_queue.cc:260] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.935917 4125 ts_tablet_manager.cc:1428] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:01:49.935942 3824 heartbeater.cc:503] Master 127.1.217.126:37641 was elected leader, sending a full tablet report...
I20260926 07:01:49.935983 4127 raft_consensus.cc:348] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.936132 4127 raft_consensus.cc:374] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:49.936169 4127 raft_consensus.cc:729] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Initialized, Role: FOLLOWER
I20260926 07:01:49.936282 4127 consensus_queue.cc:260] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:49.936575 4127 ts_tablet_manager.cc:1428] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:01:49.936592 3964 heartbeater.cc:503] Master 127.1.217.126:37641 was elected leader, sending a full tablet report...
W20260926 07:01:49.990108 3965 tablet.cc:2369] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:50.004197 4130 raft_consensus.cc:482] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:50.004330 4130 raft_consensus.cc:504] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.004802 4130 leader_election.cc:288] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241)
I20260926 07:01:50.008628 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.005050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60223 (local address 127.1.217.66:34241)
0926 07:01:50.005090 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.005092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.005236 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:01:50.005269 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.005270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.005279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.005329 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:50.005486 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.006223 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.007672 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.008229 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.008304 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.008313 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.008494 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.008535 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:50.008670 3997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.004965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.1.217.67:48413)
0926 07:01:50.005024 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.005048 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.005095 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.005246 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.005247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.005248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:50.005352 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.005354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.006329 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.006330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.007578 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.007582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.008321 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.008322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.008411 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.008415 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.008478 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.008521 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":69}
I20260926 07:01:50.008705 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.005019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:48413 (local address 127.1.217.65:42813)
0926 07:01:50.005067 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.005070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.005080 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:50.005146 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.005147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.005161 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.005238 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:50.005425 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.006215 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.007671 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.008228 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.008304 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.008332 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.008583 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.008635 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:50.008924 4133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.004981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:34241 (local address 127.1.217.67:60223)
0926 07:01:50.005157 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.005166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.005220 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.005348 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.005348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.005350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:50.005420 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.005421 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.006329 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.006330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.007582 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.007583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.008347 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.008348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.008413 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.008416 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.008477 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.008521 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":30,"thread_start_us":114,"threads_started":1}
I20260926 07:01:50.009055 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f751ab63a55544e486c67f4aa97894ad" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" is_pre_election: true
I20260926 07:01:50.009212 3778 raft_consensus.cc:2454] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 0.
I20260926 07:01:50.009289 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f751ab63a55544e486c67f4aa97894ad" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0" is_pre_election: true
I20260926 07:01:50.009434 3918 raft_consensus.cc:2454] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 0.
I20260926 07:01:50.009521 3990 leader_election.cc:302] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c931b92d1c614b2fb52898369fa7455e, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:50.009644 4130 raft_consensus.cc:2781] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:50.009711 4130 raft_consensus.cc:482] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:50.009729 4130 raft_consensus.cc:3037] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.010426 4130 raft_consensus.cc:504] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.010564 4130 leader_election.cc:288] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 election: Requested vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241)
I20260926 07:01:50.010720 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f751ab63a55544e486c67f4aa97894ad" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
I20260926 07:01:50.010749 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f751ab63a55544e486c67f4aa97894ad" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0"
I20260926 07:01:50.010823 3778 raft_consensus.cc:3037] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.010823 3918 raft_consensus.cc:3037] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.011796 3918 raft_consensus.cc:2454] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 1.
I20260926 07:01:50.011915 3778 raft_consensus.cc:2454] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 1.
I20260926 07:01:50.011989 3989 leader_election.cc:302] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, c931b92d1c614b2fb52898369fa7455e; no voters:
I20260926 07:01:50.012089 4130 raft_consensus.cc:2781] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:50.012238 4130 raft_consensus.cc:686] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 1 LEADER]: Becoming Leader. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Running, Role: LEADER
I20260926 07:01:50.012346 4130 consensus_queue.cc:237] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.013389 3621 catalog_manager.cc:5495] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e reported cstate change: term changed from 0 to 1, leader changed from <none> to c931b92d1c614b2fb52898369fa7455e (127.1.217.67). New cstate: current_term: 1 leader_uuid: "c931b92d1c614b2fb52898369fa7455e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } health_report { overall_health: HEALTHY } } }
Sep 26 07:01:50 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406109, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:50.039780 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.033431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37641 (local address 127.0.0.1:35982)
0926 07:01:50.033619 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.033630 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.033692 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.033808 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.033809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.034121 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:50.034164 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.034166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.034776 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.034777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.035345 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.035346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.035660 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.035661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.035702 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.036185 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:50.036189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:50.037555 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:50.038889 (+ 1334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:50.038895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:50.038897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:50.039088 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:50.039256 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:50.039257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:50.039257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:50.039310 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:50.039504 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:50.039509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:50.039552 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.039605 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.039691 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":121,"threads_started":1}
I20260926 07:01:50.039829 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.033483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35982 (local address 127.1.217.126:37641)
0926 07:01:50.033554 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.033557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.033697 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:01:50.033740 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.033741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.033750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.033791 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:50.034233 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.034707 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.035418 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.035596 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.035646 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.037629 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:50.037630 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:50.037631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:50.037636 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:50.038820 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:50.039152 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:50.039153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:50.039153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:50.039200 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:50.039361 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:50.039361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:50.039362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:50.039454 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:50.039501 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.039637 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.039737 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":42}
I20260926 07:01:50.042004 3621 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:35982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:01:50.042213 3621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:50.043794 3739 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf474e788644e18b388d62ec825652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae3de3c9b29e4469be9988e272bcd167]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:50.043805 3884 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf474e788644e18b388d62ec825652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae3de3c9b29e4469be9988e272bcd167]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:50.043987 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf474e788644e18b388d62ec825652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.043991 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf474e788644e18b388d62ec825652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.044114 4038 tablet_service.cc:1467] Processing CreateTablet for tablet eeaf474e788644e18b388d62ec825652 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ae3de3c9b29e4469be9988e272bcd167]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:01:50.044255 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eeaf474e788644e18b388d62ec825652. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.045343 4125 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: Bootstrap starting.
I20260926 07:01:50.045542 4127 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: Bootstrap starting.
I20260926 07:01:50.045702 4124 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: Bootstrap starting.
I20260926 07:01:50.046094 4124 tablet_bootstrap.cc:654] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.046132 4125 tablet_bootstrap.cc:654] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.046254 4127 tablet_bootstrap.cc:654] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.046720 4124 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: No bootstrap required, opened a new log
I20260926 07:01:50.046775 4124 ts_tablet_manager.cc:1397] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:50.046942 4124 raft_consensus.cc:348] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.047021 4127 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: No bootstrap required, opened a new log
I20260926 07:01:50.047034 4124 raft_consensus.cc:374] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.047058 4124 raft_consensus.cc:729] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.047065 4127 ts_tablet_manager.cc:1397] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:01:50.047123 4124 consensus_queue.cc:260] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.047245 4124 ts_tablet_manager.cc:1428] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:50.047204 4127 raft_consensus.cc:348] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.047295 4127 raft_consensus.cc:374] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.047322 4127 raft_consensus.cc:729] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.047295 4134 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.047369 4127 consensus_queue.cc:260] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.047418 4134 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.047461 4127 ts_tablet_manager.cc:1428] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:50.047518 4132 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.047580 4132 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.047998 4125 tablet_bootstrap.cc:492] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: No bootstrap required, opened a new log
I20260926 07:01:50.048058 4125 ts_tablet_manager.cc:1397] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:01:50.048202 4125 raft_consensus.cc:348] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.048261 4125 raft_consensus.cc:374] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.048275 4125 raft_consensus.cc:729] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.048331 4125 consensus_queue.cc:260] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.048422 4125 ts_tablet_manager.cc:1428] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:50.048482 4131 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.048553 4131 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 07:01:50.085278 3825 tablet.cc:2369] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:50.126513 4131 raft_consensus.cc:482] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:50.126646 4131 raft_consensus.cc:504] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.127022 4131 leader_election.cc:288] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:50.129909 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.127375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:47971 (local address 127.1.217.67:39407)
0926 07:01:50.127423 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.127425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.127434 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:50.127462 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.127463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.127473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.127545 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:50.127701 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.128168 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129195 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.129552 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129644 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.129653 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.129798 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.129832 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:50.129940 4121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.127295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55263 (local address 127.1.217.66:34241)
0926 07:01:50.127352 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.127354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.127513 (+ 159us) server_negotiation.cc:407] Connection header received
0926 07:01:50.127591 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.127592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.127601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.127671 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:50.127818 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.128287 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129235 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.129590 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129670 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.129678 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.129836 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.129870 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:01:50.129951 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.127218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:39407 (local address 127.1.217.65:47971)
0926 07:01:50.127338 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.127349 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.127406 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.127562 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.127563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.127566 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:50.127622 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.127624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.128250 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.128251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.129101 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.129103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.129643 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.129720 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.129724 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.129791 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.129832 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 07:01:50.129969 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.127218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:34241 (local address 127.1.217.65:55263)
0926 07:01:50.127428 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.127437 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.127526 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.127673 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.127674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.127676 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:50.127748 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.127750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.128377 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.128378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.129150 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.129152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.129680 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.129681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.129756 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.129759 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.129819 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.129860 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":13,"thread_start_us":50,"threads_started":1}
I20260926 07:01:50.130190 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf474e788644e18b388d62ec825652" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0" is_pre_election: true
I20260926 07:01:50.130240 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf474e788644e18b388d62ec825652" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e" is_pre_election: true
I20260926 07:01:50.130288 3918 raft_consensus.cc:2454] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 0.
I20260926 07:01:50.130342 4058 raft_consensus.cc:2454] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 0.
I20260926 07:01:50.130450 3709 leader_election.cc:302] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:50.130606 4131 raft_consensus.cc:2781] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:50.130661 4131 raft_consensus.cc:482] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:50.130688 4131 raft_consensus.cc:3037] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.131287 4131 raft_consensus.cc:504] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.131425 4131 leader_election.cc:288] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 election: Requested vote from peers 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:50.131578 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf474e788644e18b388d62ec825652" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e"
I20260926 07:01:50.131654 4058 raft_consensus.cc:3037] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.131625 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eeaf474e788644e18b388d62ec825652" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0"
I20260926 07:01:50.131706 3918 raft_consensus.cc:3037] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.132275 3918 raft_consensus.cc:2454] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 1.
I20260926 07:01:50.132334 4058 raft_consensus.cc:2454] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 1.
I20260926 07:01:50.132438 3709 leader_election.cc:302] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:50.132512 4131 raft_consensus.cc:2781] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:50.132627 4131 raft_consensus.cc:686] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 LEADER]: Becoming Leader. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Running, Role: LEADER
I20260926 07:01:50.132716 4131 consensus_queue.cc:237] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.133119 4147 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: TxnStatusTablet state changed. Reason: New leader cd7c3eb04f3c42889d336db9f0fd7f26. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.133221 4147 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:50.133392 4149 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:01:50.133517 4149 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:01:50.133484 3620 catalog_manager.cc:5495] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } health_report { overall_health: UNKNOWN } } }
W20260926 07:01:50.139848 4105 tablet.cc:2369] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:01:50.142016 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.139059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.1.217.65:42813)
0926 07:01:50.139108 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.139111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.139119 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:50.139208 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.139208 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.139242 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.139329 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.139451 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.140000 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.141186 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.141465 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.141519 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.141732 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.141817 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.141876 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.141951 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.141968 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:01:50.142099 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.138991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.0.0.1:36802)
0926 07:01:50.139060 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.139071 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.139120 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.139337 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.139338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.139339 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:50.139383 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.139386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.140109 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.140111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.141108 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.141112 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.141546 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.141547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.141621 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.141638 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.141883 (+ 245us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.141891 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.141949 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.142000 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:01:50.143802 4058 raft_consensus.cc:1261] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Refusing update from remote peer cd7c3eb04f3c42889d336db9f0fd7f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.143803 3918 raft_consensus.cc:1261] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Refusing update from remote peer cd7c3eb04f3c42889d336db9f0fd7f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.144105 4131 consensus_queue.cc:1035] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:50.144258 4147 consensus_queue.cc:1035] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:50.145450 4134 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: TxnStatusTablet state changed. Reason: New leader cd7c3eb04f3c42889d336db9f0fd7f26. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.145533 4134 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.146332 4152 mvcc.cc:205] Tried to move back new op lower bound from 7333503427146690560 to 7333503427104264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503427146690560}]
I20260926 07:01:50.146683 4131 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.146770 4131 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:50.147181 4132 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: TxnStatusTablet state changed. Reason: New leader cd7c3eb04f3c42889d336db9f0fd7f26. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.147276 4132 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.147333 4153 mvcc.cc:205] Tried to move back new op lower bound from 7333503427146690560 to 7333503427104264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503427146690560}]
I20260926 07:01:50.147408 4134 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.147486 4134 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P c931b92d1c614b2fb52898369fa7455e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.148595 4131 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.148676 4131 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P cd7c3eb04f3c42889d336db9f0fd7f26: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:01:50.149117 4132 tablet_replica.cc:439] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } } }
I20260926 07:01:50.149194 4132 tablet_replica.cc:442] T eeaf474e788644e18b388d62ec825652 P 0c9614e8e2c343c7a590b6406ff051f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:01:50.149248 4155 mvcc.cc:205] Tried to move back new op lower bound from 7333503427146690560 to 7333503427104264192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503427146690560}]
I20260926 07:01:50.155833 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.153225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.1.217.67:39407)
0926 07:01:50.153279 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.153280 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.153288 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:50.153377 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.153378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.153394 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.153460 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.153604 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.154176 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.155046 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.155249 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.155322 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.155504 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.155613 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.155708 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.155733 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.155752 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:01:50.160966 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.158260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.1.217.67:38877)
0926 07:01:50.158321 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.158333 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.158383 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.158573 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.158575 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.158576 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:50.158633 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.158634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.159521 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.159521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.160076 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.160077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.160537 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.160538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.160588 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.160601 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.160829 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.160831 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.160883 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.160916 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 07:01:50.160981 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.158338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:38877 (local address 127.1.217.65:42813)
0926 07:01:50.158380 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.158383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.158391 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:50.158439 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.158440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.158451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.158512 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.158719 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.159443 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.160161 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.160454 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.160522 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.160696 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.160727 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.160804 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.160892 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.160911 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:50.163385 3918 raft_consensus.cc:1261] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Refusing update from remote peer c931b92d1c614b2fb52898369fa7455e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.163430 3778 raft_consensus.cc:1261] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Refusing update from remote peer c931b92d1c614b2fb52898369fa7455e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.163663 4134 consensus_queue.cc:1035] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:50.163784 4130 consensus_queue.cc:1035] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:50.172315 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.169679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55573 (local address 127.1.217.67:39407)
0926 07:01:50.169717 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.169719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.169871 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:01:50.169938 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.169939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.169946 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.169990 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.170123 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.170585 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.171730 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.171902 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.171953 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.172141 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.172163 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.172195 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.172252 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.172265 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:50.172386 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.169644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:39407 (local address 127.1.217.65:55573)
0926 07:01:50.169816 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.169827 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.169882 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.169993 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.169994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.169995 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:50.170067 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.170068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.170658 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.170659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.171649 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.171652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.171980 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.171981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.172053 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.172067 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.172201 (+ 134us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.172203 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.172250 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.172288 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260926 07:01:50.181057 3620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:50.181177 3620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:50.181919 3620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Sending DeleteTablet for 3 replicas of tablet f751ab63a55544e486c67f4aa97894ad
I20260926 07:01:50.182240 4038 tablet_service.cc:1514] Processing DeleteTablet for tablet f751ab63a55544e486c67f4aa97894ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:50 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:56264
I20260926 07:01:50.182328 3884 tablet_service.cc:1514] Processing DeleteTablet for tablet f751ab63a55544e486c67f4aa97894ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:50 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33266
I20260926 07:01:50.182545 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet f751ab63a55544e486c67f4aa97894ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:50 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36790
I20260926 07:01:50.182775 4173 tablet_replica.cc:330] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: stopping tablet replica
I20260926 07:01:50.182912 4173 raft_consensus.cc:2229] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:50.183019 4173 raft_consensus.cc:2258] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:50.182958 3620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:50.183089 3620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:50.183454 4172 tablet_replica.cc:330] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: stopping tablet replica
I20260926 07:01:50.183557 4172 raft_consensus.cc:2229] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:50.183622 4172 raft_consensus.cc:2258] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:50.183771 4171 tablet_replica.cc:330] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: stopping tablet replica
I20260926 07:01:50.183858 4171 raft_consensus.cc:2229] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:50.183949 4171 raft_consensus.cc:2258] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:50.184378 4171 ts_tablet_manager.cc:1905] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:50.184378 4173 ts_tablet_manager.cc:1905] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:50.184494 3884 tablet_service.cc:1467] Processing CreateTablet for tablet aeb350f8a5684b7fafe7dc232df2669a (DEFAULT_TABLE table=test-table [id=4d34809a6a174d57808cbe6ece14f192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:50.184623 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb350f8a5684b7fafe7dc232df2669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.185953 4127 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Bootstrap starting.
I20260926 07:01:50.186300 4171 ts_tablet_manager.cc:1918] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:50.186374 4171 log.cc:1196] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/f751ab63a55544e486c67f4aa97894ad
I20260926 07:01:50.186388 4127 tablet_bootstrap.cc:654] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.186641 4171 ts_tablet_manager.cc:1939] T f751ab63a55544e486c67f4aa97894ad P c931b92d1c614b2fb52898369fa7455e: Deleting consensus metadata
I20260926 07:01:50.184396 3739 tablet_service.cc:1467] Processing CreateTablet for tablet aeb350f8a5684b7fafe7dc232df2669a (DEFAULT_TABLE table=test-table [id=4d34809a6a174d57808cbe6ece14f192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:50.186867 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb350f8a5684b7fafe7dc232df2669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.187135 4173 ts_tablet_manager.cc:1918] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:50.187224 4173 log.cc:1196] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/f751ab63a55544e486c67f4aa97894ad
I20260926 07:01:50.187465 4173 ts_tablet_manager.cc:1939] T f751ab63a55544e486c67f4aa97894ad P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting consensus metadata
I20260926 07:01:50.187856 3606 catalog_manager.cc:4841] TS cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813): tablet f751ab63a55544e486c67f4aa97894ad (table test-table [id=711068f9a3cc4276a4620ea8099be229]) successfully deleted
I20260926 07:01:50.188017 4125 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Bootstrap starting.
I20260926 07:01:50.188089 3607 catalog_manager.cc:4841] TS c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407): tablet f751ab63a55544e486c67f4aa97894ad (table test-table [id=711068f9a3cc4276a4620ea8099be229]) successfully deleted
I20260926 07:01:50.188418 4125 tablet_bootstrap.cc:654] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.188563 4172 ts_tablet_manager.cc:1905] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:50.188890 4038 tablet_service.cc:1467] Processing CreateTablet for tablet aeb350f8a5684b7fafe7dc232df2669a (DEFAULT_TABLE table=test-table [id=4d34809a6a174d57808cbe6ece14f192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:50.188988 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeb350f8a5684b7fafe7dc232df2669a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:50.189990 4124 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Bootstrap starting.
I20260926 07:01:50.190094 4172 ts_tablet_manager.cc:1918] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:50.190165 4172 log.cc:1196] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/f751ab63a55544e486c67f4aa97894ad
I20260926 07:01:50.190382 4124 tablet_bootstrap.cc:654] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:50.190405 4172 ts_tablet_manager.cc:1939] T f751ab63a55544e486c67f4aa97894ad P 0c9614e8e2c343c7a590b6406ff051f0: Deleting consensus metadata
I20260926 07:01:50.190793 3605 catalog_manager.cc:4841] TS 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241): tablet f751ab63a55544e486c67f4aa97894ad (table test-table [id=711068f9a3cc4276a4620ea8099be229]) successfully deleted
I20260926 07:01:50.191088 4124 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: No bootstrap required, opened a new log
I20260926 07:01:50.191154 4124 ts_tablet_manager.cc:1397] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:50.191299 4124 raft_consensus.cc:348] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.191372 4124 raft_consensus.cc:374] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.191401 4124 raft_consensus.cc:729] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.191438 4124 consensus_queue.cc:260] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.191534 4124 ts_tablet_manager.cc:1428] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:50.191736 4125 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: No bootstrap required, opened a new log
I20260926 07:01:50.191789 4125 ts_tablet_manager.cc:1397] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:01:50.191952 4125 raft_consensus.cc:348] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.192041 4125 raft_consensus.cc:374] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.192077 4125 raft_consensus.cc:729] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.192118 4125 consensus_queue.cc:260] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.192215 4125 ts_tablet_manager.cc:1428] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:50.204581 4159 raft_consensus.cc:482] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:50.204663 4159 raft_consensus.cc:504] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.204830 4159 leader_election.cc:288] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:50.205027 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb350f8a5684b7fafe7dc232df2669a" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0" is_pre_election: true
I20260926 07:01:50.205184 4127 tablet_bootstrap.cc:492] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: No bootstrap required, opened a new log
I20260926 07:01:50.205235 4127 ts_tablet_manager.cc:1397] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.001s
W20260926 07:01:50.205344 3709 leader_election.cc:341] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241): Illegal state: must be running to vote when last-logged opid is not known
I20260926 07:01:50.205374 4127 raft_consensus.cc:348] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.205433 4127 raft_consensus.cc:374] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:50.205452 4127 raft_consensus.cc:729] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Initialized, Role: FOLLOWER
I20260926 07:01:50.205497 4127 consensus_queue.cc:260] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.205595 4127 ts_tablet_manager.cc:1428] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:50.205961 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb350f8a5684b7fafe7dc232df2669a" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e" is_pre_election: true
I20260926 07:01:50.206077 4058 raft_consensus.cc:2454] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 0.
I20260926 07:01:50.206230 3711 leader_election.cc:302] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c931b92d1c614b2fb52898369fa7455e, cd7c3eb04f3c42889d336db9f0fd7f26; no voters: 0c9614e8e2c343c7a590b6406ff051f0
I20260926 07:01:50.206333 4159 raft_consensus.cc:2781] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:50.206367 4159 raft_consensus.cc:482] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:50.206386 4159 raft_consensus.cc:3037] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.207229 4159 raft_consensus.cc:504] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.207369 4159 leader_election.cc:288] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 election: Requested vote from peers 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:50.207561 3918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb350f8a5684b7fafe7dc232df2669a" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0"
I20260926 07:01:50.207559 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeb350f8a5684b7fafe7dc232df2669a" candidate_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e"
I20260926 07:01:50.207620 3918 raft_consensus.cc:3037] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.207631 4058 raft_consensus.cc:3037] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:50.208325 3918 raft_consensus.cc:2454] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 1.
I20260926 07:01:50.208436 4058 raft_consensus.cc:2454] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd7c3eb04f3c42889d336db9f0fd7f26 in term 1.
I20260926 07:01:50.208472 3709 leader_election.cc:302] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:50.208865 4159 raft_consensus.cc:2781] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:50.208981 4159 raft_consensus.cc:686] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 LEADER]: Becoming Leader. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Running, Role: LEADER
I20260926 07:01:50.209028 4159 consensus_queue.cc:237] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:50.209683 3620 catalog_manager.cc:5495] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:50.219393 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.216853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.1.217.65:42813)
0926 07:01:50.216894 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.216895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.216925 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:01:50.217002 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.217003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.217030 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.217080 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.217178 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.217693 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.218631 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.218882 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.218950 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.219108 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.219141 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.219216 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.219273 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.219292 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:01:50.223619 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.221569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:47745 (local address 127.1.217.65:42813)
0926 07:01:50.221598 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:50.221599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:50.221607 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:50.221666 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:50.221666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:50.221675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:50.221726 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:50.221820 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.222320 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.223009 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.223197 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.223277 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.223389 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.223417 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.223456 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:50.223525 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 07:01:50.223539 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:01:50.223623 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:50.221520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.1.217.65:47745)
0926 07:01:50.221556 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:50.221567 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:50.221609 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:50.221729 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:50.221729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:50.221730 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:50.221766 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.221767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.222406 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.222407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.222919 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:50.222920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:50.223276 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:50.223277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:50.223324 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:50.223329 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:50.223465 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:50.223467 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:50.223515 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:01:50.223553 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":1}
I20260926 07:01:50.225649 3918 raft_consensus.cc:1261] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Refusing update from remote peer cd7c3eb04f3c42889d336db9f0fd7f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.225839 4131 consensus_queue.cc:1035] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:50.225881 4058 raft_consensus.cc:1261] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Refusing update from remote peer cd7c3eb04f3c42889d336db9f0fd7f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:50.226073 4131 consensus_queue.cc:1035] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:01:50.231726 4152 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:01:51.751992 3620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:51.752086 3620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35974:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:51.752908 3620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Sending DeleteTablet for 3 replicas of tablet aeb350f8a5684b7fafe7dc232df2669a
I20260926 07:01:51.753201 4038 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb350f8a5684b7fafe7dc232df2669a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:56264
I20260926 07:01:51.753196 3884 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb350f8a5684b7fafe7dc232df2669a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33266
I20260926 07:01:51.753247 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet aeb350f8a5684b7fafe7dc232df2669a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36790
I20260926 07:01:51.753443 4200 tablet_replica.cc:330] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: stopping tablet replica
I20260926 07:01:51.753714 4202 tablet_replica.cc:330] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: stopping tablet replica
I20260926 07:01:51.753765 4202 raft_consensus.cc:2229] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:51.753845 4202 raft_consensus.cc:2258] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.753957 4200 raft_consensus.cc:2229] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:51.754041 4200 raft_consensus.cc:2258] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.754092 4202 ts_tablet_manager.cc:1905] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.754282 4200 ts_tablet_manager.cc:1905] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.756022 4202 ts_tablet_manager.cc:1918] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:01:51.756038 4200 ts_tablet_manager.cc:1918] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:01:51.756069 4202 log.cc:1196] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/aeb350f8a5684b7fafe7dc232df2669a
I20260926 07:01:51.756090 4200 log.cc:1196] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/aeb350f8a5684b7fafe7dc232df2669a
I20260926 07:01:51.756309 4200 ts_tablet_manager.cc:1939] T aeb350f8a5684b7fafe7dc232df2669a P 0c9614e8e2c343c7a590b6406ff051f0: Deleting consensus metadata
I20260926 07:01:51.756319 4202 ts_tablet_manager.cc:1939] T aeb350f8a5684b7fafe7dc232df2669a P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting consensus metadata
I20260926 07:01:51.756675 3606 catalog_manager.cc:4841] TS cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813): tablet aeb350f8a5684b7fafe7dc232df2669a (table test-table [id=4d34809a6a174d57808cbe6ece14f192]) successfully deleted
I20260926 07:01:51.756743 3605 catalog_manager.cc:4841] TS 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241): tablet aeb350f8a5684b7fafe7dc232df2669a (table test-table [id=4d34809a6a174d57808cbe6ece14f192]) successfully deleted
I20260926 07:01:51.756848 4201 tablet_replica.cc:330] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: stopping tablet replica
I20260926 07:01:51.756920 4201 raft_consensus.cc:2229] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:51.756963 4201 raft_consensus.cc:2258] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.757153 4201 ts_tablet_manager.cc:1905] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.758383 4201 ts_tablet_manager.cc:1918] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:01:51.758440 4201 log.cc:1196] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/aeb350f8a5684b7fafe7dc232df2669a
I20260926 07:01:51.758635 4201 ts_tablet_manager.cc:1939] T aeb350f8a5684b7fafe7dc232df2669a P c931b92d1c614b2fb52898369fa7455e: Deleting consensus metadata
I20260926 07:01:51.759054 3607 catalog_manager.cc:4841] TS c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407): tablet aeb350f8a5684b7fafe7dc232df2669a (table test-table [id=4d34809a6a174d57808cbe6ece14f192]) successfully deleted
Sep 26 07:01:51 dist-test-slave-t1mc krb5kdc[3576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:01:51 dist-test-slave-t1mc krb5kdc[3576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406111, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:51.776541 4210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.769773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.1.217.126:37641)
0926 07:01:51.769914 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.769916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.769957 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:01:51.770050 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.770051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.770064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.770127 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:51.770676 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.771170 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.771883 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.772086 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.772145 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.774289 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:51.774290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:51.774291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:51.774296 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:51.775513 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:51.775827 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:51.775828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:51.775829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:51.775891 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:51.776066 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:51.776067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:51.776067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:51.776164 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:51.776241 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.776318 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.776398 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 07:01:51.777982 3620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:01:51.778128 3620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:51.779352 3739 tablet_service.cc:1467] Processing CreateTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 (DEFAULT_TABLE table=test-table [id=7b33c6bb28654023adb8dc4bd4451453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.779373 3884 tablet_service.cc:1467] Processing CreateTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 (DEFAULT_TABLE table=test-table [id=7b33c6bb28654023adb8dc4bd4451453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.779397 4038 tablet_service.cc:1467] Processing CreateTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 (DEFAULT_TABLE table=test-table [id=7b33c6bb28654023adb8dc4bd4451453]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.779489 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7148b9a3ca0a47f4bd51b58e64904006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.779489 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7148b9a3ca0a47f4bd51b58e64904006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.779547 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7148b9a3ca0a47f4bd51b58e64904006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.780485 4211 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Bootstrap starting.
I20260926 07:01:51.780917 4211 tablet_bootstrap.cc:654] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.780999 4212 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Bootstrap starting.
I20260926 07:01:51.781375 4212 tablet_bootstrap.cc:654] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.781591 4211 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: No bootstrap required, opened a new log
I20260926 07:01:51.781649 4211 ts_tablet_manager.cc:1397] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:51.781736 4213 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Bootstrap starting.
I20260926 07:01:51.781770 4211 raft_consensus.cc:348] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.781843 4211 raft_consensus.cc:374] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.781876 4211 raft_consensus.cc:729] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.781952 4211 consensus_queue.cc:260] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.782121 4211 ts_tablet_manager.cc:1428] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:51.782130 4213 tablet_bootstrap.cc:654] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.782480 4212 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: No bootstrap required, opened a new log
I20260926 07:01:51.782546 4212 ts_tablet_manager.cc:1397] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:01:51.782688 4212 raft_consensus.cc:348] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.782788 4212 raft_consensus.cc:374] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.782812 4212 raft_consensus.cc:729] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.782857 4212 consensus_queue.cc:260] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.782929 4213 tablet_bootstrap.cc:492] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: No bootstrap required, opened a new log
I20260926 07:01:51.782979 4213 ts_tablet_manager.cc:1397] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:51.783013 4212 ts_tablet_manager.cc:1428] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:51.783082 4213 raft_consensus.cc:348] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.783150 4213 raft_consensus.cc:374] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.783171 4213 raft_consensus.cc:729] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.783221 4213 consensus_queue.cc:260] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.783344 4213 ts_tablet_manager.cc:1428] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:51.813205 4218 raft_consensus.cc:482] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:51.813319 4218 raft_consensus.cc:504] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.813701 4218 leader_election.cc:288] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:51.816972 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.813960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:35469 (local address 127.1.217.67:39407)
0926 07:01:51.814129 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.814131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.814140 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:01:51.814221 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.814222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.814240 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.814317 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:51.814477 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.814965 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816222 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.816612 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816700 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.816709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.816855 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.816892 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260926 07:01:51.816985 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.813911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:39407 (local address 127.1.217.66:35469)
0926 07:01:51.814082 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:51.814094 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:51.814151 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:51.814321 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:51.814322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:51.814324 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:51.814408 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.814409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.815047 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.815048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.816138 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.816140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.816699 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.816759 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.816792 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 07:01:51.816849 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:01:51.816898 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":109,"thread_start_us":97,"threads_started":1}
I20260926 07:01:51.817073 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.813917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50407 (local address 127.1.217.65:42813)
0926 07:01:51.814089 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.814091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.814101 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:01:51.814204 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.814205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.814216 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.814308 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:01:51.814467 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.814957 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816238 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.816623 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816709 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.816719 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.816947 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.816988 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":111,"threads_started":1}
I20260926 07:01:51.817128 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.813849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.1.217.66:50407)
0926 07:01:51.814040 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:51.814053 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:51.814113 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:51.814311 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:51.814312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:51.814314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:01:51.814391 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.814392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.815041 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.815042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.816153 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.816155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.816707 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.816708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.816826 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.816829 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:01:51.816940 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:01:51.816980 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":31,"thread_start_us":111,"threads_started":1}
I20260926 07:01:51.817414 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7148b9a3ca0a47f4bd51b58e64904006" candidate_uuid: "0c9614e8e2c343c7a590b6406ff051f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" is_pre_election: true
I20260926 07:01:51.817431 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7148b9a3ca0a47f4bd51b58e64904006" candidate_uuid: "0c9614e8e2c343c7a590b6406ff051f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e" is_pre_election: true
I20260926 07:01:51.817518 4058 raft_consensus.cc:2454] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9614e8e2c343c7a590b6406ff051f0 in term 0.
I20260926 07:01:51.817531 3778 raft_consensus.cc:2454] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c9614e8e2c343c7a590b6406ff051f0 in term 0.
I20260926 07:01:51.817739 3850 leader_election.cc:302] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:51.817888 4218 raft_consensus.cc:2781] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:51.817936 4218 raft_consensus.cc:482] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:51.817960 4218 raft_consensus.cc:3037] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:51.818660 4218 raft_consensus.cc:504] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.818794 4218 leader_election.cc:288] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [CANDIDATE]: Term 1 election: Requested vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407)
I20260926 07:01:51.818974 4058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7148b9a3ca0a47f4bd51b58e64904006" candidate_uuid: "0c9614e8e2c343c7a590b6406ff051f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c931b92d1c614b2fb52898369fa7455e"
I20260926 07:01:51.818984 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7148b9a3ca0a47f4bd51b58e64904006" candidate_uuid: "0c9614e8e2c343c7a590b6406ff051f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
I20260926 07:01:51.819077 3778 raft_consensus.cc:3037] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:51.819077 4058 raft_consensus.cc:3037] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:51.819677 4058 raft_consensus.cc:2454] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9614e8e2c343c7a590b6406ff051f0 in term 1.
I20260926 07:01:51.819793 3778 raft_consensus.cc:2454] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c9614e8e2c343c7a590b6406ff051f0 in term 1.
I20260926 07:01:51.819957 3850 leader_election.cc:302] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9614e8e2c343c7a590b6406ff051f0, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:51.820079 4218 raft_consensus.cc:2781] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:51.820233 4218 raft_consensus.cc:686] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Running, Role: LEADER
I20260926 07:01:51.820307 4218 consensus_queue.cc:237] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.821090 3621 catalog_manager.cc:5495] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "0c9614e8e2c343c7a590b6406ff051f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } health_report { overall_health: UNKNOWN } } }
I20260926 07:01:51.831151 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.828822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.1.217.66:34241)
0926 07:01:51.828963 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.828965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.828973 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:01:51.829016 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.829017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.829033 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.829105 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:51.829200 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.829750 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.830444 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.830646 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.830699 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.830829 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:51.830927 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:51.830980 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.831018 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.831034 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":101,"threads_started":1}
I20260926 07:01:51.835839 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.833750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:47231 (local address 127.1.217.65:42813)
0926 07:01:51.833813 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.833814 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.833879 (+ 65us) server_negotiation.cc:407] Connection header received
0926 07:01:51.833935 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.833936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.833944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.833993 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:51.834097 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.834536 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.835305 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.835484 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.835532 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.835665 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:51.835688 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:51.835724 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.835772 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.835787 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:01:51.835870 4229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.833621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.1.217.66:47231)
0926 07:01:51.833825 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:51.833835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:51.833895 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:51.834002 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:51.834002 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:51.834003 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:51.834049 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.834050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.834599 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.834600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.835239 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.835241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.835543 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.835544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.835590 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.835601 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:51.835736 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:51.835738 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:51.835771 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 07:01:51.835804 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":88,"threads_started":1}
I20260926 07:01:51.837944 4058 raft_consensus.cc:1261] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Refusing update from remote peer 0c9614e8e2c343c7a590b6406ff051f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:51.838095 3778 raft_consensus.cc:1261] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Refusing update from remote peer 0c9614e8e2c343c7a590b6406ff051f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:51.838212 4223 consensus_queue.cc:1035] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:51.838316 4218 consensus_queue.cc:1035] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:51.845906 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.843548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:57869 (local address 127.1.217.66:34241)
0926 07:01:51.843650 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:51.843652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:51.843693 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:01:51.843766 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:51.843767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:51.843777 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:51.843820 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:51.843925 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.844379 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.845204 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.845444 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.845532 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.845684 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:51.845717 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:51.845764 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:51.845827 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:01:51.845859 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 07:01:51.845934 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:51.843502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:34241 (local address 127.1.217.65:57869)
0926 07:01:51.843645 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:51.843657 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:51.843708 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:51.843831 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:51.843832 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:51.843833 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:01:51.843876 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.843878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.844464 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.844464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.845112 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:51.845114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:51.845538 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:51.845540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:51.845592 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:51.845596 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:01:51.845781 (+ 185us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:01:51.845783 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:01:51.845826 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 07:01:51.845880 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":101,"threads_started":1}
I20260926 07:01:51.852123 3620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:51.852222 3620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:51.852869 3620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Sending DeleteTablet for 3 replicas of tablet 7148b9a3ca0a47f4bd51b58e64904006
I20260926 07:01:51.853164 4038 tablet_service.cc:1514] Processing DeleteTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:56264
I20260926 07:01:51.853202 3884 tablet_service.cc:1514] Processing DeleteTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33266
I20260926 07:01:51.853286 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet 7148b9a3ca0a47f4bd51b58e64904006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:51 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36790
I20260926 07:01:51.853298 4201 tablet_replica.cc:330] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: stopping tablet replica
I20260926 07:01:51.853359 4201 raft_consensus.cc:2229] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:51.853404 4201 raft_consensus.cc:2258] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.853469 4200 tablet_replica.cc:330] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: stopping tablet replica
I20260926 07:01:51.853433 3620 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 07:01:51.853525 4200 raft_consensus.cc:2229] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 LEADER]: Raft consensus shutting down.
W20260926 07:01:51.853574 3620 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:01:51.853595 4200 raft_consensus.cc:2258] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.853686 4202 tablet_replica.cc:330] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: stopping tablet replica
I20260926 07:01:51.853739 4202 raft_consensus.cc:2229] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:51.853781 4202 raft_consensus.cc:2258] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:51.853972 4200 ts_tablet_manager.cc:1905] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.854374 4202 ts_tablet_manager.cc:1905] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.855157 4038 tablet_service.cc:1467] Processing CreateTablet for tablet c33340e9744b457597a3d0b07c0cf09c (DEFAULT_TABLE table=test-table [id=8ea4aa3d0b4b4f988899cff412fc2e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.855264 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33340e9744b457597a3d0b07c0cf09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.855614 4200 ts_tablet_manager.cc:1918] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:51.855661 4200 log.cc:1196] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/7148b9a3ca0a47f4bd51b58e64904006
I20260926 07:01:51.855872 4200 ts_tablet_manager.cc:1939] T 7148b9a3ca0a47f4bd51b58e64904006 P 0c9614e8e2c343c7a590b6406ff051f0: Deleting consensus metadata
I20260926 07:01:51.856003 4202 ts_tablet_manager.cc:1918] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:51.856046 4202 log.cc:1196] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/7148b9a3ca0a47f4bd51b58e64904006
I20260926 07:01:51.856261 4202 ts_tablet_manager.cc:1939] T 7148b9a3ca0a47f4bd51b58e64904006 P cd7c3eb04f3c42889d336db9f0fd7f26: Deleting consensus metadata
I20260926 07:01:51.856555 4201 ts_tablet_manager.cc:1905] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:51.856590 4211 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Bootstrap starting.
I20260926 07:01:51.856807 3605 catalog_manager.cc:4841] TS 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241): tablet 7148b9a3ca0a47f4bd51b58e64904006 (table test-table [id=7b33c6bb28654023adb8dc4bd4451453]) successfully deleted
I20260926 07:01:51.856839 3884 tablet_service.cc:1467] Processing CreateTablet for tablet c33340e9744b457597a3d0b07c0cf09c (DEFAULT_TABLE table=test-table [id=8ea4aa3d0b4b4f988899cff412fc2e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.856926 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33340e9744b457597a3d0b07c0cf09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.856951 4211 tablet_bootstrap.cc:654] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.857963 4201 ts_tablet_manager.cc:1918] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:01:51.858035 4201 log.cc:1196] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/7148b9a3ca0a47f4bd51b58e64904006
I20260926 07:01:51.858084 4212 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Bootstrap starting.
I20260926 07:01:51.858246 4201 ts_tablet_manager.cc:1939] T 7148b9a3ca0a47f4bd51b58e64904006 P c931b92d1c614b2fb52898369fa7455e: Deleting consensus metadata
I20260926 07:01:51.858449 4212 tablet_bootstrap.cc:654] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.858558 3607 catalog_manager.cc:4841] TS c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407): tablet 7148b9a3ca0a47f4bd51b58e64904006 (table test-table [id=7b33c6bb28654023adb8dc4bd4451453]) successfully deleted
I20260926 07:01:51.858736 3606 catalog_manager.cc:4841] TS cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813): tablet 7148b9a3ca0a47f4bd51b58e64904006 (table test-table [id=7b33c6bb28654023adb8dc4bd4451453]) successfully deleted
I20260926 07:01:51.858762 3739 tablet_service.cc:1467] Processing CreateTablet for tablet c33340e9744b457597a3d0b07c0cf09c (DEFAULT_TABLE table=test-table [id=8ea4aa3d0b4b4f988899cff412fc2e30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:01:51.858853 3739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33340e9744b457597a3d0b07c0cf09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:51.859202 4212 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: No bootstrap required, opened a new log
I20260926 07:01:51.859221 4211 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: No bootstrap required, opened a new log
I20260926 07:01:51.859244 4212 ts_tablet_manager.cc:1397] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:51.859258 4211 ts_tablet_manager.cc:1397] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:01:51.859376 4211 raft_consensus.cc:348] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.859436 4211 raft_consensus.cc:374] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.859402 4212 raft_consensus.cc:348] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.859460 4212 raft_consensus.cc:374] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.859458 4211 raft_consensus.cc:729] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.859481 4212 raft_consensus.cc:729] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9614e8e2c343c7a590b6406ff051f0, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.859517 4212 consensus_queue.cc:260] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.859519 4211 consensus_queue.cc:260] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.859604 4211 ts_tablet_manager.cc:1428] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:51.859625 4212 ts_tablet_manager.cc:1428] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:51.860011 4213 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26: Bootstrap starting.
I20260926 07:01:51.860376 4213 tablet_bootstrap.cc:654] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:51.861061 4213 tablet_bootstrap.cc:492] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26: No bootstrap required, opened a new log
I20260926 07:01:51.861114 4213 ts_tablet_manager.cc:1397] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:51.861239 4213 raft_consensus.cc:348] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.861308 4213 raft_consensus.cc:374] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:51.861331 4213 raft_consensus.cc:729] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7c3eb04f3c42889d336db9f0fd7f26, State: Initialized, Role: FOLLOWER
I20260926 07:01:51.861377 4213 consensus_queue.cc:260] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:51.861476 4213 ts_tablet_manager.cc:1428] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:52.003829 4216 raft_consensus.cc:482] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:01:52.003973 4216 raft_consensus.cc:504] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:52.004132 4216 leader_election.cc:288] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241)
I20260926 07:01:52.004804 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33340e9744b457597a3d0b07c0cf09c" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" is_pre_election: true
I20260926 07:01:52.004853 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33340e9744b457597a3d0b07c0cf09c" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0" is_pre_election: true
I20260926 07:01:52.004935 3778 raft_consensus.cc:2454] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 0.
I20260926 07:01:52.004949 3917 raft_consensus.cc:2454] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 0.
I20260926 07:01:52.005090 3990 leader_election.cc:302] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c931b92d1c614b2fb52898369fa7455e, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:52.005199 4216 raft_consensus.cc:2781] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:01:52.005255 4216 raft_consensus.cc:482] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:01:52.005276 4216 raft_consensus.cc:3037] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:52.005951 4216 raft_consensus.cc:504] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:52.006079 4216 leader_election.cc:288] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 election: Requested vote from peers cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813), 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241)
I20260926 07:01:52.006222 3778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33340e9744b457597a3d0b07c0cf09c" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26"
I20260926 07:01:52.006218 3917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33340e9744b457597a3d0b07c0cf09c" candidate_uuid: "c931b92d1c614b2fb52898369fa7455e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9614e8e2c343c7a590b6406ff051f0"
I20260926 07:01:52.006299 3778 raft_consensus.cc:3037] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:52.006310 3917 raft_consensus.cc:3037] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:52.006875 3778 raft_consensus.cc:2454] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 1.
I20260926 07:01:52.007027 3917 raft_consensus.cc:2454] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c931b92d1c614b2fb52898369fa7455e in term 1.
I20260926 07:01:52.007047 3990 leader_election.cc:302] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c931b92d1c614b2fb52898369fa7455e, cd7c3eb04f3c42889d336db9f0fd7f26; no voters:
I20260926 07:01:52.007190 4216 raft_consensus.cc:2781] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:52.007308 4216 raft_consensus.cc:686] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 1 LEADER]: Becoming Leader. State: Replica: c931b92d1c614b2fb52898369fa7455e, State: Running, Role: LEADER
I20260926 07:01:52.007383 4216 consensus_queue.cc:237] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } }
I20260926 07:01:52.007972 3621 catalog_manager.cc:5495] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e reported cstate change: term changed from 0 to 1, leader changed from <none> to c931b92d1c614b2fb52898369fa7455e (127.1.217.67). New cstate: current_term: 1 leader_uuid: "c931b92d1c614b2fb52898369fa7455e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c931b92d1c614b2fb52898369fa7455e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 39407 } health_report { overall_health: HEALTHY } } }
I20260926 07:01:52.046190 4222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:52.043983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56280 (local address 127.1.217.67:39407)
0926 07:01:52.044027 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:52.044029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:52.044036 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:01:52.044057 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:52.044058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:52.044070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:52.044114 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:01:52.044226 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:52.044704 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:52.045441 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:52.045659 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:52.045735 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:52.045905 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:01:52.045940 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:01:52.046013 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:52.046078 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 07:01:52.046101 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:01:52.048600 3778 raft_consensus.cc:1261] T c33340e9744b457597a3d0b07c0cf09c P cd7c3eb04f3c42889d336db9f0fd7f26 [term 1 FOLLOWER]: Refusing update from remote peer c931b92d1c614b2fb52898369fa7455e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:52.048611 3917 raft_consensus.cc:1261] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Refusing update from remote peer c931b92d1c614b2fb52898369fa7455e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:01:52.048830 4239 consensus_queue.cc:1035] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd7c3eb04f3c42889d336db9f0fd7f26" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:01:52.048936 4216 consensus_queue.cc:1035] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9614e8e2c343c7a590b6406ff051f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 34241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:01:52.052973 4185 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 07:01:53.569365 3620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:53.569473 3620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:01:53.570253 3620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad47ea6d9ee451f91fcdcc539673c95: Sending DeleteTablet for 3 replicas of tablet c33340e9744b457597a3d0b07c0cf09c
I20260926 07:01:53.570542 3884 tablet_service.cc:1514] Processing DeleteTablet for tablet c33340e9744b457597a3d0b07c0cf09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33266
I20260926 07:01:53.570583 3739 tablet_service.cc:1514] Processing DeleteTablet for tablet c33340e9744b457597a3d0b07c0cf09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36790
I20260926 07:01:53.570667 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3693
I20260926 07:01:53.571197 4267 tablet_replica.cc:330] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: stopping tablet replica
I20260926 07:01:53.571291 4267 raft_consensus.cc:2229] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:01:53.571364 4267 raft_consensus.cc:2258] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:53.571615 4267 ts_tablet_manager.cc:1905] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:53.573261 4038 tablet_service.cc:1514] Processing DeleteTablet for tablet c33340e9744b457597a3d0b07c0cf09c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:01:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:56264
I20260926 07:01:53.573586 4267 ts_tablet_manager.cc:1918] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:01:53.573653 4267 log.cc:1196] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/c33340e9744b457597a3d0b07c0cf09c
I20260926 07:01:53.573691 4269 tablet_replica.cc:330] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: stopping tablet replica
I20260926 07:01:53.573756 4269 raft_consensus.cc:2229] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:01:53.573830 4269 raft_consensus.cc:2258] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:01:53.573900 4267 ts_tablet_manager.cc:1939] T c33340e9744b457597a3d0b07c0cf09c P 0c9614e8e2c343c7a590b6406ff051f0: Deleting consensus metadata
I20260926 07:01:53.574095 4269 ts_tablet_manager.cc:1905] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:01:53.574429 3605 catalog_manager.cc:4841] TS 0c9614e8e2c343c7a590b6406ff051f0 (127.1.217.66:34241): tablet c33340e9744b457597a3d0b07c0cf09c (table test-table [id=8ea4aa3d0b4b4f988899cff412fc2e30]) successfully deleted
I20260926 07:01:53.575459 4269 ts_tablet_manager.cc:1918] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 07:01:53.575507 4269 log.cc:1196] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/c33340e9744b457597a3d0b07c0cf09c
I20260926 07:01:53.575722 4269 ts_tablet_manager.cc:1939] T c33340e9744b457597a3d0b07c0cf09c P c931b92d1c614b2fb52898369fa7455e: Deleting consensus metadata
I20260926 07:01:53.576058 3607 catalog_manager.cc:4841] TS c931b92d1c614b2fb52898369fa7455e (127.1.217.67:39407): tablet c33340e9744b457597a3d0b07c0cf09c (table test-table [id=8ea4aa3d0b4b4f988899cff412fc2e30]) successfully deleted
W20260926 07:01:53.578569 3606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:42813 (error 108)
I20260926 07:01:53.578810 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3833
I20260926 07:01:53.584281 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3973
I20260926 07:01:53.584363 4271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:53.584063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:42813 (local address 127.0.0.1:36820)
0926 07:01:53.584230 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:53.584300 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
W20260926 07:01:53.585011 3606 catalog_manager.cc:4584] TS cd7c3eb04f3c42889d336db9f0fd7f26 (127.1.217.65:42813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c33340e9744b457597a3d0b07c0cf09c: Network error: Client connection negotiation failed: client connection to 127.1.217.65:42813: connect: Connection refused (error 111)
I20260926 07:01:53.590119 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 3589
2026-09-26T07:01:53Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5463 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:53 dist-test-slave-t1mc krb5kdc[4274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:53 dist-test-slave-t1mc krb5kdc[4274](info): set up 2 sockets
Sep 26 07:01:53 dist-test-slave-t1mc krb5kdc[4274](info): commencing operation
krb5kdc: starting...
W20260926 07:01:54.664608 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:54 dist-test-slave-t1mc krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:54.779057 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46711
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:46711
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:54.857753 4287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:54.858007 4287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:54.858048 4287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:54.859212 4287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:54.859283 4287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:54.859306 4287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:54.859328 4287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:54.859356 4287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:54.860883 4287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:46711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:54.861163 4287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:54.861398 4287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:54.866389 4292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:54.866415 4293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:54.866478 4287 server_base.cc:1034] running on GCE node
W20260926 07:01:54.866389 4295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:54.866889 4287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:54.867225 4287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:54.868414 4287 hybrid_clock.cc:648] HybridClock initialized: now 1790406114868396 us; error 37 us; skew 500 ppm
Sep 26 07:01:54 dist-test-slave-t1mc krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406114, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:54.871050 4287 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:54.871500 4287 webserver.cc:460] Webserver started at http://127.1.217.126:40319/ using document root <none> and password file <none>
I20260926 07:01:54.871709 4287 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:54.871752 4287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:54.871842 4287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:54.872929 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "7af4818ff1fb4056a605988179ec7301"
format_stamp: "Formatted at 2026-09-26 07:01:54 on dist-test-slave-t1mc"
server_key: "87ea0cf450aafdac6593de9b7b36a089"
server_key_iv: "0aac297720688b9dd21a04cba1eb85a3"
server_key_version: "encryptionkey@0"
I20260926 07:01:54.873299 4287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "7af4818ff1fb4056a605988179ec7301"
format_stamp: "Formatted at 2026-09-26 07:01:54 on dist-test-slave-t1mc"
server_key: "87ea0cf450aafdac6593de9b7b36a089"
server_key_iv: "0aac297720688b9dd21a04cba1eb85a3"
server_key_version: "encryptionkey@0"
I20260926 07:01:54.874583 4287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:01:54.875365 4302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:54.875581 4287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:54.875705 4287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "7af4818ff1fb4056a605988179ec7301"
format_stamp: "Formatted at 2026-09-26 07:01:54 on dist-test-slave-t1mc"
server_key: "87ea0cf450aafdac6593de9b7b36a089"
server_key_iv: "0aac297720688b9dd21a04cba1eb85a3"
server_key_version: "encryptionkey@0"
I20260926 07:01:54.875792 4287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:54.895520 4287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:54.897526 4287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:54.897715 4287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:54.901553 4287 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:46711
I20260926 07:01:54.901602 4354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:46711 every 8 connection(s)
I20260926 07:01:54.901965 4287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:54.902817 4355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:54.903648 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4287
I20260926 07:01:54.903724 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:54.903924 1893 external_mini_cluster.cc:1428] Setting key adc026de7a80d7864fb9f4b1511c8aa3
I20260926 07:01:54.905776 4355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: Bootstrap starting.
I20260926 07:01:54.906399 4355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:54.906713 4355 log.cc:824] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:54.907547 4355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: No bootstrap required, opened a new log
I20260926 07:01:54.909760 4355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } }
I20260926 07:01:54.909904 4355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:54.909927 4355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af4818ff1fb4056a605988179ec7301, State: Initialized, Role: FOLLOWER
I20260926 07:01:54.910075 4355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } }
I20260926 07:01:54.910148 4355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:54.910198 4355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:54.910247 4355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:54.910859 4355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } }
I20260926 07:01:54.910983 4355 leader_election.cc:302] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7af4818ff1fb4056a605988179ec7301; no voters:
I20260926 07:01:54.911214 4355 leader_election.cc:288] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:54.911242 4360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:54.911352 4360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [term 1 LEADER]: Becoming Leader. State: Replica: 7af4818ff1fb4056a605988179ec7301, State: Running, Role: LEADER
I20260926 07:01:54.911491 4355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 07:01:54 dist-test-slave-t1mc krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:54.911502 4360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } }
I20260926 07:01:54.911938 4361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7af4818ff1fb4056a605988179ec7301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } } }
I20260926 07:01:54.912082 4361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:54.912757 4362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7af4818ff1fb4056a605988179ec7301. Latest consensus state: current_term: 1 leader_uuid: "7af4818ff1fb4056a605988179ec7301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7af4818ff1fb4056a605988179ec7301" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46711 } } }
I20260926 07:01:54.912891 4362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301 [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:54.913677 4376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:54.913735 4376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:54.913856 4369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:54.914039 4369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:54.915794 4369 catalog_manager.cc:1347] Generated new cluster ID: ea69bbbd4add40f0802f8d3ef6839ebd
I20260926 07:01:54.915900 4369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:54.916052 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:54.904502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.1.217.126:46711)
0926 07:01:54.904726 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:54.904735 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:54.904792 (+ 57us) server_negotiation.cc:407] Connection header received
0926 07:01:54.905450 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:54.905459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:54.905525 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:54.905706 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:54.906368 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:54.907793 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:54.908527 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:54.909180 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:54.909328 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:54.912242 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:54.912246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:54.912247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:54.912260 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:54.914335 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:54.914732 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:54.914735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:54.914736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:54.914839 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:54.915085 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:54.915086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:54.915086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:54.915390 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:54.915527 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:54.915662 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:01:54.915815 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":112,"threads_started":1}
I20260926 07:01:54.944427 4369 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:54.945037 4369 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:54.950474 4369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7af4818ff1fb4056a605988179ec7301: Generated new TSK 0
I20260926 07:01:54.950668 4369 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:54.979804 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:55.061656 4381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:55.061866 4381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:55.061889 4381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:55.063181 4381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:55.063225 4381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:55.063252 4381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:55.064841 4381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:55.065083 4381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:55.065308 4381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:55.070449 4386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:55.070462 4387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:55.070708 4381 server_base.cc:1034] running on GCE node
W20260926 07:01:55.070462 4389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:55.070972 4381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:55.071219 4381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:55.072367 4381 hybrid_clock.cc:648] HybridClock initialized: now 1790406115072346 us; error 34 us; skew 500 ppm
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:55.075074 4381 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:55.075541 4381 webserver.cc:460] Webserver started at http://127.1.217.65:34671/ using document root <none> and password file <none>
I20260926 07:01:55.075765 4381 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:55.075831 4381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:55.075981 4381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:55.077099 4381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "98beeb51e7a64e12a1f72f9f9c8c0239"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "9c9bc2a44602cf09f8fadbc01b8b251b"
server_key_iv: "a378f317194ee153a744b749ce9d34da"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.077458 4381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "98beeb51e7a64e12a1f72f9f9c8c0239"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "9c9bc2a44602cf09f8fadbc01b8b251b"
server_key_iv: "a378f317194ee153a744b749ce9d34da"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.078855 4381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:55.079566 4396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.079727 4381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:55.079818 4381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "98beeb51e7a64e12a1f72f9f9c8c0239"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "9c9bc2a44602cf09f8fadbc01b8b251b"
server_key_iv: "a378f317194ee153a744b749ce9d34da"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.079895 4381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:55.098953 4381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:55.100873 4381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:55.101043 4381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:55.101298 4381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:55.101624 4381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:55.101679 4381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.101724 4381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:55.101771 4381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.107492 4381 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:39647
I20260926 07:01:55.107585 4509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:39647 every 8 connection(s)
I20260926 07:01:55.107921 4381 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:55.114502 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4381
I20260926 07:01:55.114578 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:55.114822 1893 external_mini_cluster.cc:1428] Setting key b6b1e88e6c28e523d2d0f1ea31a10f31
I20260926 07:01:55.117916 4512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.108625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46711 (local address 127.1.217.65:50937)
0926 07:01:55.108827 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:55.108869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:55.109355 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:55.109576 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:55.109580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:55.110170 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:55.110374 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.110379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.111099 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.111100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.112092 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.112095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.112420 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.112420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.112556 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.113146 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:55.113156 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:55.114609 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:55.116468 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.116475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.116477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.116843 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.117060 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.117061 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.117061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.117158 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.117426 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:55.117431 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:55.117542 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 07:01:55.117643 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:01:55.117760 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260926 07:01:55.117944 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.108676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:50937 (local address 127.1.217.126:46711)
0926 07:01:55.108743 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:55.108745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:55.108961 (+ 216us) server_negotiation.cc:407] Connection header received
0926 07:01:55.109437 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:55.109438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:55.109448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:55.109510 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:55.110501 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.111023 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.112162 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.112352 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.112406 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.114753 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:55.114755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:55.114755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:55.114763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:55.116384 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.116922 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.116924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.116924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.116992 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.117226 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.117227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.117227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.117366 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:55.117429 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:55.117704 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:01:55.117798 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:01:55.118410 4510 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46711
I20260926 07:01:55.118512 4510 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:55.118706 4510 heartbeater.cc:511] Master 127.1.217.126:46711 requested a full tablet report, sending...
I20260926 07:01:55.119221 4319 ts_manager.cc:194] Registered new tserver with Master: 98beeb51e7a64e12a1f72f9f9c8c0239 (127.1.217.65:39647)
I20260926 07:01:55.119844 4319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:50937
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:55.137993 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:55.217849 4515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:55.218075 4515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:55.218110 4515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:55.219416 4515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:55.219504 4515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:55.219551 4515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:55.221129 4515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:55.221379 4515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:55.221617 4515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:55.226522 4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:55.226569 4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:55.226569 4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:55.226801 4515 server_base.cc:1034] running on GCE node
I20260926 07:01:55.227059 4515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:55.227298 4515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:55.228438 4515 hybrid_clock.cc:648] HybridClock initialized: now 1790406115228428 us; error 30 us; skew 500 ppm
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:55.230875 4515 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:55.231329 4515 webserver.cc:460] Webserver started at http://127.1.217.66:43787/ using document root <none> and password file <none>
I20260926 07:01:55.231547 4515 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:55.231613 4515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:55.231766 4515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:55.232834 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "27f1c3c8bfcd44608f16996059321b2f"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "f563b80152c24bd35dd6e32ed6092694"
server_key_iv: "23741a00359f3872edd72bb6ad4619b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.233206 4515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "27f1c3c8bfcd44608f16996059321b2f"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "f563b80152c24bd35dd6e32ed6092694"
server_key_iv: "23741a00359f3872edd72bb6ad4619b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.234519 4515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:55.235209 4530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.235385 4515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:55.235478 4515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "27f1c3c8bfcd44608f16996059321b2f"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "f563b80152c24bd35dd6e32ed6092694"
server_key_iv: "23741a00359f3872edd72bb6ad4619b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.235564 4515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:55.258766 4515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:55.260609 4515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:55.260821 4515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:55.261080 4515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:55.261395 4515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:55.261449 4515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.261490 4515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:55.261518 4515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.267350 4515 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:42463
I20260926 07:01:55.267400 4643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:42463 every 8 connection(s)
I20260926 07:01:55.267702 4515 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:55.272670 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4515
I20260926 07:01:55.272759 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:55.272969 1893 external_mini_cluster.cc:1428] Setting key df49922b78e861f977fcc904fc230cbe
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:55.277559 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.268374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:33613 (local address 127.1.217.126:46711)
0926 07:01:55.268434 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:55.268436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:55.268649 (+ 213us) server_negotiation.cc:407] Connection header received
0926 07:01:55.269218 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:55.269219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:55.269228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:55.269284 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:55.270172 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.270647 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.271633 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.271809 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.271857 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.274390 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:55.274391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:55.274391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:55.274397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:55.276055 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.276558 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.276559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.276559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.276610 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.276906 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.276907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.276907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.277007 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:55.277058 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:55.277422 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 07:01:55.277497 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:01:55.277819 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.268344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46711 (local address 127.1.217.66:33613)
0926 07:01:55.268519 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:55.268560 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:55.269130 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:55.269336 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:55.269340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:55.269895 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:55.270098 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.270103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.270744 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.270746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.271558 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.271560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.271878 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.271879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.271976 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.272406 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:55.272414 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:55.274304 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:55.276140 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.276146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.276148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.276474 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.276674 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.276675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.276675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.276842 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.277072 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:55.277078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:55.277243 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 07:01:55.277357 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:01:55.277544 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260926 07:01:55.278344 4644 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46711
I20260926 07:01:55.278452 4644 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:55.278666 4644 heartbeater.cc:511] Master 127.1.217.126:46711 requested a full tablet report, sending...
I20260926 07:01:55.279036 4319 ts_manager.cc:194] Registered new tserver with Master: 27f1c3c8bfcd44608f16996059321b2f (127.1.217.66:42463)
I20260926 07:01:55.279439 4319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:33613
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:55.298897 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:43495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:55.377818 4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:55.378065 4649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:55.378093 4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:55.379386 4649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:55.379458 4649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:55.379489 4649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:55.380977 4649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46711
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:55.381225 4649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:55.381445 4649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:55.386409 4655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:55.386457 4657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:55.386456 4654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:55.386654 4649 server_base.cc:1034] running on GCE node
I20260926 07:01:55.386830 4649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:55.387024 4649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:55.388175 4649 hybrid_clock.cc:648] HybridClock initialized: now 1790406115388161 us; error 36 us; skew 500 ppm
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:55.390585 4649 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:55.390985 4649 webserver.cc:460] Webserver started at http://127.1.217.67:40489/ using document root <none> and password file <none>
I20260926 07:01:55.391217 4649 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:55.391304 4649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:55.391456 4649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:55.392448 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "d5716e805b634e0ca0ab98d70729f342"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "6ff10d3251ff85ba0e6dd5366503e297"
server_key_iv: "d9cb6434e0261a3a6007f31592d9e8fe"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.393010 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "d5716e805b634e0ca0ab98d70729f342"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "6ff10d3251ff85ba0e6dd5366503e297"
server_key_iv: "d9cb6434e0261a3a6007f31592d9e8fe"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.394335 4649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:55.395022 4664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.395195 4649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:55.395293 4649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "d5716e805b634e0ca0ab98d70729f342"
format_stamp: "Formatted at 2026-09-26 07:01:55 on dist-test-slave-t1mc"
server_key: "6ff10d3251ff85ba0e6dd5366503e297"
server_key_iv: "d9cb6434e0261a3a6007f31592d9e8fe"
server_key_version: "encryptionkey@0"
I20260926 07:01:55.395373 4649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:55.407163 4649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:55.409624 4649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:55.409781 4649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:55.410020 4649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:55.410326 4649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:55.410359 4649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.410413 4649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:55.410440 4649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:55.416205 4649 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:43595
I20260926 07:01:55.416296 4777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:43595 every 8 connection(s)
I20260926 07:01:55.416580 4649 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406115, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:55.423305 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4649
I20260926 07:01:55.423399 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:55.423621 1893 external_mini_cluster.cc:1428] Setting key 45db27187bd5af902447ff1c4f29c8bd
I20260926 07:01:55.425407 4358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.417278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:56761 (local address 127.1.217.126:46711)
0926 07:01:55.417324 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:55.417326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:55.417527 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:01:55.417986 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:55.417987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:55.417997 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:55.418045 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:55.418896 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.419372 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.420318 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.420494 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.420543 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.422602 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:55.422603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:55.422603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:55.422610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:55.424184 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.424621 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.424622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.424622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.424672 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:55.424936 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:55.424937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:55.424937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:55.425034 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:55.425092 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:55.425291 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 07:01:55.425347 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:01:55.425559 4780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:55.417276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46711 (local address 127.1.217.67:56761)
0926 07:01:55.417431 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:55.417463 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:55.417903 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:55.418107 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:55.418110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:55.418648 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:55.418824 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.418829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.419468 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.419469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.420244 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:55.420245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:55.420562 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:55.420563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:55.420687 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:55.421246 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:55.421253 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:55.422518 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:55.424256 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.424262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.424264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.424552 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.424714 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:55.424715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:55.424715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:55.424883 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:55.425086 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:55.425089 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:55.425190 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 07:01:55.425265 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:01:55.425374 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260926 07:01:55.425985 4778 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46711
I20260926 07:01:55.426086 4778 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:55.426256 4778 heartbeater.cc:511] Master 127.1.217.126:46711 requested a full tablet report, sending...
I20260926 07:01:55.426599 4319 ts_manager.cc:194] Registered new tserver with Master: d5716e805b634e0ca0ab98d70729f342 (127.1.217.67:43595)
I20260926 07:01:55.426919 4319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:56761
I20260926 07:01:55.435190 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:01:55.445917 4358 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:01:55.445039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.1.217.126:46711)
0926 07:01:55.445120 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:55.445122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:55.445251 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:01:55.445293 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:55.445294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:55.445302 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:55.445358 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:55.445847 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40550: BlockingRecv error: recv got EOF from 127.0.0.1:40550 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:01:55.446725 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4381
I20260926 07:01:55.451117 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4515
I20260926 07:01:55.455225 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4649
I20260926 07:01:55.459234 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4287
2026-09-26T07:01:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1867 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4790](info): set up 2 sockets
Sep 26 07:01:55 dist-test-slave-t1mc krb5kdc[4790](info): commencing operation
krb5kdc: starting...
W20260926 07:01:56.484438 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:56.599159 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43841
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:34693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:56.681955 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:56.682156 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:56.682178 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:56.683377 4803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:56.683414 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:56.683429 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:56.683442 4803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:56.683455 4803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:56.684959 4803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:34693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:56.685199 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:56.685446 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:56.690163 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:56.690186 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:56.690204 4803 server_base.cc:1034] running on GCE node
W20260926 07:01:56.690187 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:56.690632 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:56.690879 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:56.692059 4803 hybrid_clock.cc:648] HybridClock initialized: now 1790406116692049 us; error 29 us; skew 500 ppm
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406116, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:56.694588 4803 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:56.695056 4803 webserver.cc:460] Webserver started at http://127.1.217.126:46155/ using document root <none> and password file <none>
I20260926 07:01:56.695297 4803 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:56.695341 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:56.695482 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:56.696550 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "379822f1d55543b9ae8b5049bcfecb83"
format_stamp: "Formatted at 2026-09-26 07:01:56 on dist-test-slave-t1mc"
server_key: "c3a96b57375ec07a3319561ca1fc0cb3"
server_key_iv: "bf4887ac43758c4c19cad44a0fefce57"
server_key_version: "encryptionkey@0"
I20260926 07:01:56.696942 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "379822f1d55543b9ae8b5049bcfecb83"
format_stamp: "Formatted at 2026-09-26 07:01:56 on dist-test-slave-t1mc"
server_key: "c3a96b57375ec07a3319561ca1fc0cb3"
server_key_iv: "bf4887ac43758c4c19cad44a0fefce57"
server_key_version: "encryptionkey@0"
I20260926 07:01:56.698266 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:56.699080 4818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:56.699288 4803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:01:56.699383 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "379822f1d55543b9ae8b5049bcfecb83"
format_stamp: "Formatted at 2026-09-26 07:01:56 on dist-test-slave-t1mc"
server_key: "c3a96b57375ec07a3319561ca1fc0cb3"
server_key_iv: "bf4887ac43758c4c19cad44a0fefce57"
server_key_version: "encryptionkey@0"
I20260926 07:01:56.699460 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:56.715708 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:56.717707 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:56.717861 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:56.721936 4803 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43841
I20260926 07:01:56.721959 4870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43841 every 8 connection(s)
I20260926 07:01:56.722311 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:56.723161 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:56.723918 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4803
I20260926 07:01:56.724035 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:56.724211 1893 external_mini_cluster.cc:1428] Setting key e983417d1d74ea5019337c368bd62699
I20260926 07:01:56.726332 4871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: Bootstrap starting.
I20260926 07:01:56.726918 4871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:56.727198 4871 log.cc:824] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:56.727842 4871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: No bootstrap required, opened a new log
I20260926 07:01:56.729624 4871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } }
I20260926 07:01:56.729775 4871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:56.729821 4871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379822f1d55543b9ae8b5049bcfecb83, State: Initialized, Role: FOLLOWER
I20260926 07:01:56.729990 4871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } }
I20260926 07:01:56.730083 4871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:56.730127 4871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:56.730176 4871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:56.730834 4871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } }
I20260926 07:01:56.730942 4871 leader_election.cc:302] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 379822f1d55543b9ae8b5049bcfecb83; no voters:
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:56.731106 4871 leader_election.cc:288] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:56.731313 4877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:56.731379 4871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:56.731523 4877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [term 1 LEADER]: Becoming Leader. State: Replica: 379822f1d55543b9ae8b5049bcfecb83, State: Running, Role: LEADER
I20260926 07:01:56.731619 4877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } }
I20260926 07:01:56.732234 4876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "379822f1d55543b9ae8b5049bcfecb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } } }
I20260926 07:01:56.732239 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 379822f1d55543b9ae8b5049bcfecb83. Latest consensus state: current_term: 1 leader_uuid: "379822f1d55543b9ae8b5049bcfecb83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379822f1d55543b9ae8b5049bcfecb83" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43841 } } }
I20260926 07:01:56.732350 4876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:56.732358 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83 [sys.catalog]: This master's current role is: LEADER
W20260926 07:01:56.733376 4891 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:01:56.733777 4891 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:01:56.733839 4887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:56.734010 4887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:56.735247 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:56.724733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.1.217.126:43841)
0926 07:01:56.724941 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:56.724953 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:56.724984 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:01:56.725388 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:56.725395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:56.725483 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:56.725598 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:56.726407 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:56.727501 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:56.728308 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:56.728723 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:56.728922 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:56.731570 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:56.731574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:56.731575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:56.731586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:56.733483 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:56.733911 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:56.733914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:56.733915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:56.734015 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:56.734249 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:56.734250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:56.734251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:56.734489 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:56.734647 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:56.734927 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 07:01:56.735064 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
I20260926 07:01:56.736181 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:01:56.741197 4887 catalog_manager.cc:1347] Generated new cluster ID: 8a49f2c4a98f4db1b0dc300ae4af299f
I20260926 07:01:56.741238 4887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:01:56.754778 4887 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:56.755476 4887 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:56.759701 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:56.755697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.1.217.126:43841)
0926 07:01:56.755734 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:56.755736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:56.755808 (+ 72us) server_negotiation.cc:407] Connection header received
0926 07:01:56.755857 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:56.755857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:56.755868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:56.755951 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:56.756316 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:56.757103 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:56.758143 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:56.758469 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:56.758540 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:56.759247 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:56.759248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:56.759248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:01:56.759456 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:56.759474 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:01:56.759480 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:56.759541 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:56.759600 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:01:56.759625 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:01:56.761071 4887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 379822f1d55543b9ae8b5049bcfecb83: Generated new TSK 0
I20260926 07:01:56.761230 4887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:01:56.766084 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4803
2026-09-26T07:01:56Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1304 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4904](info): set up 2 sockets
Sep 26 07:01:56 dist-test-slave-t1mc krb5kdc[4904](info): commencing operation
krb5kdc: starting...
W20260926 07:01:57.788545 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:57 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:57.903959 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:40371
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:57.982558 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:57.982761 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:57.982784 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:57.983984 4917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:57.984028 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:57.984042 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:57.984055 4917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:57.984113 4917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:57.985595 4917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:40371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:57.985868 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:57.986109 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:57.991044 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:57.991052 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:57.991036 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:57.991075 4917 server_base.cc:1034] running on GCE node
I20260926 07:01:57.991508 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:57.991837 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:57.992997 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790406117992983 us; error 42 us; skew 500 ppm
Sep 26 07:01:57 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406117, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:57.995373 4917 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:57.995738 4917 webserver.cc:460] Webserver started at http://127.1.217.126:46537/ using document root <none> and password file <none>
I20260926 07:01:57.995982 4917 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:57.996028 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:57.996174 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:57.997280 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "a7a368dd74e04722b37ed4b746dc1f38"
format_stamp: "Formatted at 2026-09-26 07:01:57 on dist-test-slave-t1mc"
server_key: "2676fbbc1cbb3b0de9f2dca6e496bdf7"
server_key_iv: "05207d5bc5036524bede1b454a0d180f"
server_key_version: "encryptionkey@0"
I20260926 07:01:57.997627 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "a7a368dd74e04722b37ed4b746dc1f38"
format_stamp: "Formatted at 2026-09-26 07:01:57 on dist-test-slave-t1mc"
server_key: "2676fbbc1cbb3b0de9f2dca6e496bdf7"
server_key_iv: "05207d5bc5036524bede1b454a0d180f"
server_key_version: "encryptionkey@0"
I20260926 07:01:57.999002 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:57.999753 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.000029 4917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:58.000124 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "a7a368dd74e04722b37ed4b746dc1f38"
format_stamp: "Formatted at 2026-09-26 07:01:57 on dist-test-slave-t1mc"
server_key: "2676fbbc1cbb3b0de9f2dca6e496bdf7"
server_key_iv: "05207d5bc5036524bede1b454a0d180f"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.000209 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:58.033399 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:58.035403 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:58.035545 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:58.039724 4917 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:40371
I20260926 07:01:58.039794 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:40371 every 8 connection(s)
I20260926 07:01:58.040126 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:58.041162 4985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:58.043408 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38: Bootstrap starting.
I20260926 07:01:58.043982 4985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:58.044256 4985 log.cc:824] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:58.044977 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38: No bootstrap required, opened a new log
I20260926 07:01:58.046556 4985 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } }
I20260926 07:01:58.046691 4985 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:58.046741 4985 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a368dd74e04722b37ed4b746dc1f38, State: Initialized, Role: FOLLOWER
I20260926 07:01:58.046918 4985 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } }
I20260926 07:01:58.046998 4985 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:58.047047 4985 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:58.047096 4985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:58.047876 4985 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } }
I20260926 07:01:58.047991 4985 leader_election.cc:302] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7a368dd74e04722b37ed4b746dc1f38; no voters:
I20260926 07:01:58.048158 4985 leader_election.cc:288] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:58.048184 4988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:58.048332 4988 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [term 1 LEADER]: Becoming Leader. State: Replica: a7a368dd74e04722b37ed4b746dc1f38, State: Running, Role: LEADER
I20260926 07:01:58.048431 4985 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:58.048431 4988 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } }
I20260926 07:01:58.048632 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 4917
I20260926 07:01:58.048723 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:58.048755 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7a368dd74e04722b37ed4b746dc1f38. Latest consensus state: current_term: 1 leader_uuid: "a7a368dd74e04722b37ed4b746dc1f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } } }
I20260926 07:01:58.048848 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:58.048924 1893 external_mini_cluster.cc:1428] Setting key 0c5cd19636911127c3d8f68ccebc97dd
I20260926 07:01:58.049422 4989 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7a368dd74e04722b37ed4b746dc1f38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7a368dd74e04722b37ed4b746dc1f38" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40371 } } }
I20260926 07:01:58.049580 4989 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38 [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:58.049070 4993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:58.050299 4993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:58.052420 4993 catalog_manager.cc:1347] Generated new cluster ID: 8ad677152f014a34bef7b4099dc00598
I20260926 07:01:58.052474 4993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:58.060554 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.049799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.1.217.126:40371)
0926 07:01:58.049990 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:58.050000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:58.050030 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:01:58.050135 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:58.050140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:58.050218 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:58.050370 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:58.051108 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.052271 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.053328 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.053969 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.054094 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.056949 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:58.056953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:58.056954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:58.056967 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:58.058894 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.059289 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.059292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.059292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.059409 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.059620 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.059621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.059621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.059919 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:58.060058 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:58.060181 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 07:01:58.060307 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260926 07:01:58.063283 4993 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:58.063773 4993 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:58.071604 4993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7a368dd74e04722b37ed4b746dc1f38: Generated new TSK 0
I20260926 07:01:58.071815 4993 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:58.099916 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:58.180867 5011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:58.181054 5011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:58.181077 5011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:58.182196 5011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:58.182240 5011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:58.182265 5011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:01:58.183743 5011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:58.183979 5011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:58.184223 5011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:58.189245 5017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:58.189254 5016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:58.189294 5019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:58.189599 5011 server_base.cc:1034] running on GCE node
I20260926 07:01:58.189783 5011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:58.190013 5011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:58.191196 5011 hybrid_clock.cc:648] HybridClock initialized: now 1790406118191181 us; error 24 us; skew 500 ppm
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:58.193739 5011 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:01:58.194205 5011 webserver.cc:460] Webserver started at http://127.1.217.65:45401/ using document root <none> and password file <none>
I20260926 07:01:58.194469 5011 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:58.194542 5011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:58.194708 5011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:58.196163 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "8e78db9e1b264c62898a803a21f8f0e9"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "c89b6886c53aefa41dd943ac7b9df74c"
server_key_iv: "480221309fb27885a9397488e2119e58"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.196615 5011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "8e78db9e1b264c62898a803a21f8f0e9"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "c89b6886c53aefa41dd943ac7b9df74c"
server_key_iv: "480221309fb27885a9397488e2119e58"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.198478 5011 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 07:01:58.199467 5026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.199684 5011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:01:58.199787 5011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "8e78db9e1b264c62898a803a21f8f0e9"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "c89b6886c53aefa41dd943ac7b9df74c"
server_key_iv: "480221309fb27885a9397488e2119e58"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.199883 5011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:58.222570 5011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:58.224484 5011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:58.224596 5011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:58.224854 5011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:58.225188 5011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:58.225242 5011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.225294 5011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:58.225334 5011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.231120 5011 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41071
I20260926 07:01:58.231163 5139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41071 every 8 connection(s)
I20260926 07:01:58.231480 5011 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:01:58.235206 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5011
I20260926 07:01:58.235298 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:01:58.235471 1893 external_mini_cluster.cc:1428] Setting key e2b142acef10c58e37f3698651b7dd66
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:58.241187 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.232183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55147 (local address 127.1.217.126:40371)
0926 07:01:58.232249 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:58.232252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:58.232475 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:01:58.232996 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:58.232997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:58.233008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:58.233068 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:58.234017 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.234548 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.235829 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.236038 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.236124 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.238359 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:58.238360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:58.238361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:58.238369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:58.239792 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.240197 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.240198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.240199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.240260 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.240510 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.240511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.240511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.240622 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:58.240699 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:58.241016 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:01:58.241096 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:01:58.241377 5142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.232175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:40371 (local address 127.1.217.65:55147)
0926 07:01:58.232358 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:58.232406 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:58.232899 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:58.233146 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:58.233150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:58.233722 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:58.233922 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.233926 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.234628 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.234629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.235738 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.235740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.236132 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.236133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.236269 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.236822 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:58.236832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:58.238252 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:58.239884 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.239890 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.239892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.240119 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.240328 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.240329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.240329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.240441 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.240690 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:58.240696 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:58.240884 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 07:01:58.240994 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:01:58.241150 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260926 07:01:58.241888 5140 heartbeater.cc:347] Connected to a master server at 127.1.217.126:40371
I20260926 07:01:58.241989 5140 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:58.242180 5140 heartbeater.cc:511] Master 127.1.217.126:40371 requested a full tablet report, sending...
I20260926 07:01:58.242774 4949 ts_manager.cc:194] Registered new tserver with Master: 8e78db9e1b264c62898a803a21f8f0e9 (127.1.217.65:41071)
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
I20260926 07:01:58.243645 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:55147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:01:58.259495 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:58.339498 5145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:58.339697 5145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:58.339720 5145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:58.340827 5145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:58.340860 5145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:58.340885 5145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:01:58.342218 5145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:58.342397 5145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:58.342571 5145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:58.347464 5150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:58.347608 5145 server_base.cc:1034] running on GCE node
W20260926 07:01:58.347501 5151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:58.347469 5153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:58.347929 5145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:58.348120 5145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:58.349279 5145 hybrid_clock.cc:648] HybridClock initialized: now 1790406118349267 us; error 30 us; skew 500 ppm
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:58.351680 5145 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:01:58.352151 5145 webserver.cc:460] Webserver started at http://127.1.217.66:42185/ using document root <none> and password file <none>
I20260926 07:01:58.352380 5145 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:58.352447 5145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:58.352577 5145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:58.353799 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "14098c867c624cb18d1cb46a27a6ab64"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "bf578ae96b67dfd8244d71dcff759877"
server_key_iv: "3dfb5e41a11599f5ffdd6dacfb3fb8e4"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.354174 5145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "14098c867c624cb18d1cb46a27a6ab64"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "bf578ae96b67dfd8244d71dcff759877"
server_key_iv: "3dfb5e41a11599f5ffdd6dacfb3fb8e4"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.355495 5145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:01:58.356189 5160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.356350 5145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:01:58.356449 5145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "14098c867c624cb18d1cb46a27a6ab64"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "bf578ae96b67dfd8244d71dcff759877"
server_key_iv: "3dfb5e41a11599f5ffdd6dacfb3fb8e4"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.356535 5145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:58.380836 5145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:58.382947 5145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:58.383112 5145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:58.383373 5145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:58.383728 5145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:58.383785 5145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.383828 5145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:58.383868 5145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.390038 5145 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:39519
I20260926 07:01:58.390084 5273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:39519 every 8 connection(s)
I20260926 07:01:58.390388 5145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:01:58.394230 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5145
I20260926 07:01:58.394345 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:01:58.394518 1893 external_mini_cluster.cc:1428] Setting key 957da0c3414df5f20e675bf6d55fb25d
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:58.399962 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.391100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:49383 (local address 127.1.217.126:40371)
0926 07:01:58.391144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:58.391146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:58.391314 (+ 168us) server_negotiation.cc:407] Connection header received
0926 07:01:58.391793 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:58.391794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:58.391803 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:58.391866 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:58.392882 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.393365 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.394311 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.394620 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.394695 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.397208 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:58.397209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:58.397210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:58.397218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:58.398594 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.399018 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.399019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.399020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.399086 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.399331 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.399332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.399332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.399451 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:58.399526 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:58.399788 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:01:58.399868 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:01:58.400172 5276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.391030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:40371 (local address 127.1.217.66:49383)
0926 07:01:58.391191 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:58.391232 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:58.391711 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:58.391951 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:58.391955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:58.392546 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:58.392736 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.392741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.393438 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.393439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.394227 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.394228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.394798 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.394799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.394937 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.395483 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:58.395493 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:58.397112 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:58.398693 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.398700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.398702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.398929 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.399152 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.399153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.399153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.399263 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.399517 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:58.399523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:58.399660 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 07:01:58.399768 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:01:58.399932 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 07:01:58.400594 5274 heartbeater.cc:347] Connected to a master server at 127.1.217.126:40371
I20260926 07:01:58.400684 5274 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:58.400908 5274 heartbeater.cc:511] Master 127.1.217.126:40371 requested a full tablet report, sending...
I20260926 07:01:58.401309 4949 ts_manager.cc:194] Registered new tserver with Master: 14098c867c624cb18d1cb46a27a6ab64 (127.1.217.66:39519)
I20260926 07:01:58.401686 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:49383
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:01:58.419540 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:58.500293 5279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:58.500487 5279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:58.500509 5279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:58.501595 5279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:58.501631 5279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:58.501658 5279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:01:58.503101 5279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:40371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:58.503334 5279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:58.503521 5279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:01:58.508564 5285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:58.508543 5284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:58.508571 5279 server_base.cc:1034] running on GCE node
W20260926 07:01:58.508527 5287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:58.509001 5279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:58.509279 5279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:58.510454 5279 hybrid_clock.cc:648] HybridClock initialized: now 1790406118510459 us; error 46 us; skew 500 ppm
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:58.512815 5279 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:01:58.513252 5279 webserver.cc:460] Webserver started at http://127.1.217.67:45347/ using document root <none> and password file <none>
I20260926 07:01:58.513517 5279 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:58.513561 5279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:58.513726 5279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:58.514741 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "a93ee0c85ba643d3bf7a4b5d45bd7861"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "93c5965b96c97e5e849d39b5bdc35e6d"
server_key_iv: "991330a11e2f3a3fcdd0b5eec9d0209f"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.515086 5279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "a93ee0c85ba643d3bf7a4b5d45bd7861"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "93c5965b96c97e5e849d39b5bdc35e6d"
server_key_iv: "991330a11e2f3a3fcdd0b5eec9d0209f"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.516384 5279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:01:58.517146 5294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.517382 5279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:01:58.517480 5279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "a93ee0c85ba643d3bf7a4b5d45bd7861"
format_stamp: "Formatted at 2026-09-26 07:01:58 on dist-test-slave-t1mc"
server_key: "93c5965b96c97e5e849d39b5bdc35e6d"
server_key_iv: "991330a11e2f3a3fcdd0b5eec9d0209f"
server_key_version: "encryptionkey@0"
I20260926 07:01:58.517549 5279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:58.543838 5279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:58.545938 5279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:58.546048 5279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:58.546213 5279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:01:58.546540 5279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:01:58.546573 5279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.546597 5279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:01:58.546643 5279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:58.552663 5279 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:46245
I20260926 07:01:58.552704 5407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:46245 every 8 connection(s)
I20260926 07:01:58.553036 5279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:01:58.554396 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5279
I20260926 07:01:58.554493 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:01:58.554697 1893 external_mini_cluster.cc:1428] Setting key b9efbc71bce35474aeb7139f97e97447
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:58.562687 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.553693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:45323 (local address 127.1.217.126:40371)
0926 07:01:58.553733 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:58.553736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:58.554011 (+ 275us) server_negotiation.cc:407] Connection header received
0926 07:01:58.554759 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:58.554761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:58.554774 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:58.554833 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:58.555772 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.556294 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.557382 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.557588 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.557673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.559802 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:58.559803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:58.559803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:58.559809 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:58.561325 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.561738 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.561739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.561740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.561810 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.562053 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.562054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.562055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.562171 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:58.562247 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:58.562518 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:01:58.562594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:01:58.562883 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.553700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:40371 (local address 127.1.217.67:45323)
0926 07:01:58.553888 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 07:01:58.553923 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:01:58.554611 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:01:58.554918 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:01:58.554921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:01:58.555476 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:01:58.555666 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.555670 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.556401 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.556402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.557280 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:01:58.557283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.557679 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.557680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:01:58.557817 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.558340 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:01:58.558348 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:01:58.559721 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:01:58.561423 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.561430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.561432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.561647 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.561877 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:01:58.561877 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:01:58.561878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:01:58.561984 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:01:58.562238 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:01:58.562244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:01:58.562388 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 07:01:58.562497 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:01:58.562651 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260926 07:01:58.563402 5408 heartbeater.cc:347] Connected to a master server at 127.1.217.126:40371
I20260926 07:01:58.563499 5408 heartbeater.cc:464] Registering TS with master...
I20260926 07:01:58.563673 5408 heartbeater.cc:511] Master 127.1.217.126:40371 requested a full tablet report, sending...
I20260926 07:01:58.564065 4949 ts_manager.cc:194] Registered new tserver with Master: a93ee0c85ba643d3bf7a4b5d45bd7861 (127.1.217.67:46245)
I20260926 07:01:58.564421 4949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:45323
I20260926 07:01:58.566201 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406118, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:58.588341 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:58.581424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.1.217.126:40371)
0926 07:01:58.581497 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:58.581499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:58.581580 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:01:58.581687 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:58.581688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:58.581698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:58.581768 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:58.582402 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.582890 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.583671 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:58.583867 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:58.583967 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:58.586209 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:58.586210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:58.586210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:58.586217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:58.587377 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.587691 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.587692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.587692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.587740 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:58.587927 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:58.587927 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:58.587927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:58.588063 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:58.588134 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:58.588220 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 07:01:58.588273 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 07:01:58.588582 4949 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45976
I20260926 07:01:58.589015 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5011
I20260926 07:01:58.593245 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5145
I20260926 07:01:58.597218 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5279
I20260926 07:01:58.601197 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 4917
2026-09-26T07:01:58Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1837 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[5420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[5420](info): set up 2 sockets
Sep 26 07:01:58 dist-test-slave-t1mc krb5kdc[5420](info): commencing operation
krb5kdc: starting...
W20260926 07:01:59.636845 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:01:59 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:01:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:01:59.758071 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33247
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:01:59.837670 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:01:59.837903 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:01:59.837939 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:01:59.839089 5433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:01:59.839160 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:01:59.839188 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:01:59.839221 5433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:01:59.839246 5433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:01:59.840812 5433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:01:59.841081 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:01:59.841303 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:01:59.846114 5433 server_base.cc:1034] running on GCE node
W20260926 07:01:59.846084 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:59.846084 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:01:59.846081 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:01:59.846483 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:01:59.846717 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:01:59.847910 5433 hybrid_clock.cc:648] HybridClock initialized: now 1790406119847892 us; error 33 us; skew 500 ppm
Sep 26 07:01:59 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406119, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:01:59.850420 5433 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:01:59.850858 5433 webserver.cc:460] Webserver started at http://127.1.217.126:43451/ using document root <none> and password file <none>
I20260926 07:01:59.851089 5433 fs_manager.cc:359] Metadata directory not provided
I20260926 07:01:59.851135 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:01:59.851224 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:01:59.852216 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "a693a5a76ff7494d86fe220f6e69e90f"
format_stamp: "Formatted at 2026-09-26 07:01:59 on dist-test-slave-t1mc"
server_key: "40cf71039688326e73a1527aa029253f"
server_key_iv: "13892ea5b386f4af67e7460f0463a5b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:59.852502 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "a693a5a76ff7494d86fe220f6e69e90f"
format_stamp: "Formatted at 2026-09-26 07:01:59 on dist-test-slave-t1mc"
server_key: "40cf71039688326e73a1527aa029253f"
server_key_iv: "13892ea5b386f4af67e7460f0463a5b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:59.853790 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:01:59.854568 5448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:01:59.854728 5433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:01:59.854805 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "a693a5a76ff7494d86fe220f6e69e90f"
format_stamp: "Formatted at 2026-09-26 07:01:59 on dist-test-slave-t1mc"
server_key: "40cf71039688326e73a1527aa029253f"
server_key_iv: "13892ea5b386f4af67e7460f0463a5b6"
server_key_version: "encryptionkey@0"
I20260926 07:01:59.854858 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:01:59.868181 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:01:59.870429 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:01:59.870537 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:01:59.874281 5433 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33247
I20260926 07:01:59.874331 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33247 every 8 connection(s)
I20260926 07:01:59.874810 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:01:59.875455 5501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:01:59.877734 5501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f: Bootstrap starting.
I20260926 07:01:59.878301 5501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f: Neither blocks nor log segments found. Creating new log.
I20260926 07:01:59.878583 5501 log.cc:824] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:01:59.879277 5501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f: No bootstrap required, opened a new log
I20260926 07:01:59.880824 5501 raft_consensus.cc:348] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } }
I20260926 07:01:59.880978 5501 raft_consensus.cc:374] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:01:59.881021 5501 raft_consensus.cc:729] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a693a5a76ff7494d86fe220f6e69e90f, State: Initialized, Role: FOLLOWER
I20260926 07:01:59.881213 5501 consensus_queue.cc:260] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } }
I20260926 07:01:59.881312 5501 raft_consensus.cc:388] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:01:59.881357 5501 raft_consensus.cc:482] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:01:59.881407 5501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:01:59.882264 5501 raft_consensus.cc:504] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } }
I20260926 07:01:59.882380 5501 leader_election.cc:302] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a693a5a76ff7494d86fe220f6e69e90f; no voters:
I20260926 07:01:59.882586 5501 leader_election.cc:288] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:01:59.882616 5504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:01:59.882732 5504 raft_consensus.cc:686] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [term 1 LEADER]: Becoming Leader. State: Replica: a693a5a76ff7494d86fe220f6e69e90f, State: Running, Role: LEADER
I20260926 07:01:59.882747 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5433
I20260926 07:01:59.882818 5504 consensus_queue.cc:237] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } }
I20260926 07:01:59.882860 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:01:59.882845 5501 sys_catalog.cc:564] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:01:59.883035 1893 external_mini_cluster.cc:1428] Setting key 6ae55b29bca21844598b78508a030f15
I20260926 07:01:59.883150 5505 sys_catalog.cc:455] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a693a5a76ff7494d86fe220f6e69e90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } } }
I20260926 07:01:59.883234 5505 sys_catalog.cc:458] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:59.883452 5508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:01:59.884006 5508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:01:59.884227 5506 sys_catalog.cc:455] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a693a5a76ff7494d86fe220f6e69e90f. Latest consensus state: current_term: 1 leader_uuid: "a693a5a76ff7494d86fe220f6e69e90f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a693a5a76ff7494d86fe220f6e69e90f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33247 } } }
I20260926 07:01:59.884305 5506 sys_catalog.cc:458] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f [sys.catalog]: This master's current role is: LEADER
I20260926 07:01:59.886304 5508 catalog_manager.cc:1347] Generated new cluster ID: 9ac4bbaf4ed94048a96950efb37c04cb
I20260926 07:01:59.886371 5508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:01:59 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:01:59.893528 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:01:59.883703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35066 (local address 127.1.217.126:33247)
0926 07:01:59.884101 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 07:01:59.884112 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:01:59.884162 (+ 50us) server_negotiation.cc:407] Connection header received
0926 07:01:59.884292 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:01:59.884300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:01:59.884359 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:01:59.884500 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:01:59.885311 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:59.886555 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:59.887313 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:01:59.887790 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:01:59.887910 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:01:59.890260 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:01:59.890264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:01:59.890264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:01:59.890276 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:01:59.891913 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:59.892292 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:59.892295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:59.892296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:59.892385 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:01:59.892591 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:01:59.892592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:01:59.892592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:01:59.892922 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:01:59.893098 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 07:01:59.893166 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:01:59.893302 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":95,"threads_started":1}
I20260926 07:01:59.896134 5508 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:01:59.896710 5508 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:01:59.902514 5508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a693a5a76ff7494d86fe220f6e69e90f: Generated new TSK 0
I20260926 07:01:59.902712 5508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:01:59.931679 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:00.034626 5527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:00.034881 5527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:00.034907 5527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:00.036087 5527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:00.036132 5527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:00.036159 5527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:00.037777 5527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:00.038012 5527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:00.038259 5527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:02:00.043308 5527 server_base.cc:1034] running on GCE node
W20260926 07:02:00.043295 5533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:00.043233 5535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:00.043233 5532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:00.043679 5527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:00.043934 5527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:00.045146 5527 hybrid_clock.cc:648] HybridClock initialized: now 1790406120045125 us; error 38 us; skew 500 ppm
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:00.047749 5527 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:02:00.048130 5527 webserver.cc:460] Webserver started at http://127.1.217.65:32921/ using document root <none> and password file <none>
I20260926 07:02:00.048305 5527 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:00.048343 5527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:00.048429 5527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:00.049542 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "072e7dc8bb50497391f509fcd9461f2f"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "c7ce15d2fdf7579d15e5fd5409c3072a"
server_key_iv: "d9ba3b2303b94bac24bbd256504f2c2a"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.049873 5527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "072e7dc8bb50497391f509fcd9461f2f"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "c7ce15d2fdf7579d15e5fd5409c3072a"
server_key_iv: "d9ba3b2303b94bac24bbd256504f2c2a"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.051079 5527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:02:00.051754 5542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.051924 5527 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:00.052011 5527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "072e7dc8bb50497391f509fcd9461f2f"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "c7ce15d2fdf7579d15e5fd5409c3072a"
server_key_iv: "d9ba3b2303b94bac24bbd256504f2c2a"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.052064 5527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:00.080770 5527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:00.082962 5527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:00.083091 5527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:00.083307 5527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:00.083629 5527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:00.083662 5527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.083684 5527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:00.083699 5527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.089519 5527 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:46295
I20260926 07:02:00.089607 5655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:46295 every 8 connection(s)
I20260926 07:02:00.089926 5527 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:00.097339 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5527
I20260926 07:02:00.097450 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:00.097630 1893 external_mini_cluster.cc:1428] Setting key ede43ff8d7dd7db73fcfd77e23e92d00
I20260926 07:02:00.100507 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.090670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:45085 (local address 127.1.217.126:33247)
0926 07:02:00.090760 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:00.090762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:00.090873 (+ 111us) server_negotiation.cc:407] Connection header received
0926 07:02:00.091803 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:00.091804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:00.091812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:00.091863 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:00.092482 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.093069 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.094021 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.094224 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.094279 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.096300 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:00.096301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:00.096301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:00.096308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:00.097873 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.098355 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.098357 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.098357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.098422 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.099847 (+ 1425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.099848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.099848 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.099963 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:00.100036 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:00.100321 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 07:02:00.100413 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:02:00.100709 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.090582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33247 (local address 127.1.217.65:45085)
0926 07:02:00.090731 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:00.090782 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:00.091712 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:00.091916 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:00.091920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:00.092196 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:00.092389 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.092393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.093149 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.093149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.093944 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.093946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.094300 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.094301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.094406 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.094858 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:00.094866 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:00.096218 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:00.097969 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.097977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.097979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.098265 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.099637 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.099638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.099638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.099742 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.100039 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:00.100046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:00.100188 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:02:00.100300 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:02:00.100456 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260926 07:02:00.101176 5656 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33247
I20260926 07:02:00.101265 5656 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:00.101452 5656 heartbeater.cc:511] Master 127.1.217.126:33247 requested a full tablet report, sending...
I20260926 07:02:00.101986 5465 ts_manager.cc:194] Registered new tserver with Master: 072e7dc8bb50497391f509fcd9461f2f (127.1.217.65:46295)
I20260926 07:02:00.102576 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:45085
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:02:00.123139 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:00.205998 5661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:00.206229 5661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:00.206271 5661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:00.207413 5661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:00.207475 5661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:00.207520 5661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:00.209072 5661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:00.209334 5661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:00.209560 5661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:00.214357 5666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:00.214352 5669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:00.214352 5667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:00.214566 5661 server_base.cc:1034] running on GCE node
I20260926 07:02:00.214780 5661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:00.215008 5661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:00.216173 5661 hybrid_clock.cc:648] HybridClock initialized: now 1790406120216163 us; error 31 us; skew 500 ppm
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:00.218636 5661 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:02:00.219055 5661 webserver.cc:460] Webserver started at http://127.1.217.66:44305/ using document root <none> and password file <none>
I20260926 07:02:00.219269 5661 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:00.219314 5661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:00.219484 5661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:00.220507 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "6e3807f44922445c859eb6be7625acc4"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "54fc5be5bd2eea9058aa7cc1dbe59ec4"
server_key_iv: "8d47dc4235e8023ae7a97fb9f91ca010"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.220922 5661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "6e3807f44922445c859eb6be7625acc4"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "54fc5be5bd2eea9058aa7cc1dbe59ec4"
server_key_iv: "8d47dc4235e8023ae7a97fb9f91ca010"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.222378 5661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:00.223040 5676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.223242 5661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:00.223338 5661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "6e3807f44922445c859eb6be7625acc4"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "54fc5be5bd2eea9058aa7cc1dbe59ec4"
server_key_iv: "8d47dc4235e8023ae7a97fb9f91ca010"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.223420 5661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:00.239650 5661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:00.241573 5661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:00.241740 5661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:00.241977 5661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:00.242290 5661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:00.242347 5661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.242388 5661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:00.242416 5661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.248032 5661 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:45021
I20260926 07:02:00.248104 5789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:45021 every 8 connection(s)
I20260926 07:02:00.248427 5661 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:00.257800 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5661
I20260926 07:02:00.257936 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:00.258096 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.249234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:52671 (local address 127.1.217.126:33247)
0926 07:02:00.249314 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:00.249316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:00.249527 (+ 211us) server_negotiation.cc:407] Connection header received
0926 07:02:00.250462 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:00.250464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:00.250474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:00.250529 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:00.251134 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.251649 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.252631 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.252909 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.252975 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.255427 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:00.255428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:00.255429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:00.255437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:00.256692 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.257088 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.257089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.257090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.257148 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.257415 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.257416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.257416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.257532 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:00.257646 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:00.257902 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 07:02:00.257994 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:02:00.258198 1893 external_mini_cluster.cc:1428] Setting key 7ed671cf9704c0ba728056ebf1cfb4ee
I20260926 07:02:00.258260 5792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.249219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33247 (local address 127.1.217.66:52671)
0926 07:02:00.249396 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:00.249432 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:00.250355 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:00.250584 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:00.250587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:00.250851 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:00.251037 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.251041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.251732 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.251733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.252529 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.252532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.252983 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.252984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.253084 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.253826 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:00.253834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:00.255301 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:00.256781 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.256787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.256788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.257025 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.257206 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.257207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.257207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.257309 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.257595 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:00.257600 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:00.257735 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 07:02:00.257836 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:02:00.258015 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260926 07:02:00.258710 5790 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33247
I20260926 07:02:00.258816 5790 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:00.259012 5790 heartbeater.cc:511] Master 127.1.217.126:33247 requested a full tablet report, sending...
I20260926 07:02:00.259351 5465 ts_manager.cc:194] Registered new tserver with Master: 6e3807f44922445c859eb6be7625acc4 (127.1.217.66:45021)
I20260926 07:02:00.259753 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:52671
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:02:00.283099 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:00.362136 5795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:00.362346 5795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:00.362368 5795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:00.363430 5795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:00.363462 5795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:00.363488 5795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:00.364966 5795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:00.365165 5795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:00.365334 5795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:00.370044 5800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:00.370095 5803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:00.370241 5795 server_base.cc:1034] running on GCE node
W20260926 07:02:00.370253 5801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:00.370565 5795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:00.370755 5795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:00.371887 5795 hybrid_clock.cc:648] HybridClock initialized: now 1790406120371877 us; error 30 us; skew 500 ppm
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:00.374342 5795 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:02:00.374749 5795 webserver.cc:460] Webserver started at http://127.1.217.67:45649/ using document root <none> and password file <none>
I20260926 07:02:00.374971 5795 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:00.375036 5795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:00.375190 5795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:00.376209 5795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "8cd005a2c60c41c89e28526a5aa15ccf"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "d58ac93672f73e3aac700bb901a6d8a6"
server_key_iv: "e6bf2114d415fe3b8eda574017d6605e"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.376562 5795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "8cd005a2c60c41c89e28526a5aa15ccf"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "d58ac93672f73e3aac700bb901a6d8a6"
server_key_iv: "e6bf2114d415fe3b8eda574017d6605e"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.377964 5795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:00.378695 5810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.378866 5795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:00.378962 5795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "8cd005a2c60c41c89e28526a5aa15ccf"
format_stamp: "Formatted at 2026-09-26 07:02:00 on dist-test-slave-t1mc"
server_key: "d58ac93672f73e3aac700bb901a6d8a6"
server_key_iv: "e6bf2114d415fe3b8eda574017d6605e"
server_key_version: "encryptionkey@0"
I20260926 07:02:00.379045 5795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:00.401648 5795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:00.404181 5795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:00.404347 5795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:00.404582 5795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:00.404956 5795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:00.405010 5795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.405051 5795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:00.405078 5795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:00.410946 5795 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:38413
I20260926 07:02:00.411007 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:38413 every 8 connection(s)
I20260926 07:02:00.411305 5795 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:00.417548 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5795
I20260926 07:02:00.417639 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:00.417834 1893 external_mini_cluster.cc:1428] Setting key ffa0e31c58dd1410865a21932b8cf28c
I20260926 07:02:00.420197 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.411969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60427 (local address 127.1.217.126:33247)
0926 07:02:00.412043 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:00.412045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:00.412229 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:02:00.413256 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:00.413257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:00.413268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:00.413334 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:00.413933 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.414429 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.415360 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.415545 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.415611 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.417706 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:00.417708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:00.417708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:00.417716 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:00.419127 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.419475 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.419476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.419476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.419523 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.419722 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.419722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.419722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.419807 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:00.419850 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:00.420045 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 07:02:00.420097 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:02:00.420338 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.411919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33247 (local address 127.1.217.67:60427)
0926 07:02:00.412067 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:00.412107 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:00.413152 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:00.413405 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:00.413409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:00.413654 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:00.413848 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.413853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.414504 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.414505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.415280 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:00.415281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.415616 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.415617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:00.415718 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.416159 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:00.416169 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:00.417585 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:00.419195 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.419199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.419201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.419419 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.419589 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:00.419589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:00.419589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:00.419684 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:00.419855 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:00.419858 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:00.419958 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 07:02:00.420033 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:02:00.420139 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260926 07:02:00.420794 5924 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33247
I20260926 07:02:00.420903 5924 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:00.421090 5924 heartbeater.cc:511] Master 127.1.217.126:33247 requested a full tablet report, sending...
I20260926 07:02:00.421484 5465 ts_manager.cc:194] Registered new tserver with Master: 8cd005a2c60c41c89e28526a5aa15ccf (127.1.217.67:38413)
I20260926 07:02:00.421931 5465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:60427
I20260926 07:02:00.429343 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:02:00.450901 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.443775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56894 (local address 127.1.217.65:46295)
0926 07:02:00.443953 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:00.443958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:00.443968 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:00.443993 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:00.443995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:00.444008 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:00.444074 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:00.444610 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.445381 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.446069 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.446387 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.446443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.448444 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:00.448451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:00.448451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:00.448458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:00.449920 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.450219 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.450222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.450222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.450272 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.450410 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.450410 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.450411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.450515 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:00.450606 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:00.450694 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 07:02:00.450750 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:00.458781 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:00.452132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.1.217.126:33247)
0926 07:02:00.452201 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:00.452204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:00.452242 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:02:00.452318 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:00.452318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:00.452328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:00.452384 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:00.452865 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.453343 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.454069 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:00.454277 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:00.454352 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:00.456366 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:00.456367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:00.456368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:00.456375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:00.457610 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.457946 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.457947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.457948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.458002 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:00.458213 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:00.458214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:00.458215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:00.458416 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:00.458502 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:00.458598 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:02:00.458683 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 07:02:00.458977 5465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35078
I20260926 07:02:00.459304 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5527
I20260926 07:02:00.464551 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5661
I20260926 07:02:00.471102 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5795
I20260926 07:02:00.475302 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5433
2026-09-26T07:02:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1873 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5936](info): set up 2 sockets
Sep 26 07:02:00 dist-test-slave-t1mc krb5kdc[5936](info): commencing operation
krb5kdc: starting...
W20260926 07:02:01.508960 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:01 dist-test-slave-t1mc krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:02:01.635334 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39613
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:39613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:01.717752 5949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:01.717943 5949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:01.717964 5949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:01.719110 5949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:01.719152 5949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:01.719166 5949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:01.719179 5949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:01.719234 5949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:01.720685 5949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:39613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:01.720973 5949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:01.721211 5949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:01.726068 5955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:01.726075 5957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:01.726177 5949 server_base.cc:1034] running on GCE node
W20260926 07:02:01.726086 5954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:01.726475 5949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:01.726801 5949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:01.727978 5949 hybrid_clock.cc:648] HybridClock initialized: now 1790406121727967 us; error 34 us; skew 500 ppm
Sep 26 07:02:01 dist-test-slave-t1mc krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:01.730451 5949 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:02:01.730782 5949 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:01.730854 5949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:01.730983 5949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:01.732002 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "f415daa64fae4161a34e676eeb66ee11"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "61c95edcd25f8117ecde739b670cc717"
server_key_iv: "b8f51cb24ec2e5c404c152606a0c7871"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.732389 5949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "f415daa64fae4161a34e676eeb66ee11"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "61c95edcd25f8117ecde739b670cc717"
server_key_iv: "b8f51cb24ec2e5c404c152606a0c7871"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.733752 5949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:01.734515 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:01.734701 5949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:01.734804 5949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "f415daa64fae4161a34e676eeb66ee11"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "61c95edcd25f8117ecde739b670cc717"
server_key_iv: "b8f51cb24ec2e5c404c152606a0c7871"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.734885 5949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:01.752527 5949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:01.754491 5949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:01.754659 5949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:01.758567 5949 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:39613
I20260926 07:02:01.758591 6015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:39613 every 8 connection(s)
I20260926 07:02:01.758893 5949 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:01.759860 6016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:01.760847 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 5949
I20260926 07:02:01.760954 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:01.761127 1893 external_mini_cluster.cc:1428] Setting key 4be374f6f875ab3dc6f459b14d26ed3d
I20260926 07:02:01.762633 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: Bootstrap starting.
I20260926 07:02:01.763391 6016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:01.763696 6016 log.cc:824] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:01.764377 6016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: No bootstrap required, opened a new log
I20260926 07:02:01.766798 6016 raft_consensus.cc:348] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } }
I20260926 07:02:01.766984 6016 raft_consensus.cc:374] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:01.767025 6016 raft_consensus.cc:729] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f415daa64fae4161a34e676eeb66ee11, State: Initialized, Role: FOLLOWER
I20260926 07:02:01.767172 6016 consensus_queue.cc:260] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } }
I20260926 07:02:01.767261 6016 raft_consensus.cc:388] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:01.767289 6016 raft_consensus.cc:482] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:01.767333 6016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:01.767920 6016 raft_consensus.cc:504] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } }
Sep 26 07:02:01 dist-test-slave-t1mc krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:01.768007 6016 leader_election.cc:302] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f415daa64fae4161a34e676eeb66ee11; no voters:
I20260926 07:02:01.768226 6016 leader_election.cc:288] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:01.768249 6021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:01.768362 6021 raft_consensus.cc:686] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [term 1 LEADER]: Becoming Leader. State: Replica: f415daa64fae4161a34e676eeb66ee11, State: Running, Role: LEADER
I20260926 07:02:01.768452 6021 consensus_queue.cc:237] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } }
I20260926 07:02:01.768497 6016 sys_catalog.cc:564] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:01.768786 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f415daa64fae4161a34e676eeb66ee11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } } }
I20260926 07:02:01.768859 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:01.768985 6021 sys_catalog.cc:455] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f415daa64fae4161a34e676eeb66ee11. Latest consensus state: current_term: 1 leader_uuid: "f415daa64fae4161a34e676eeb66ee11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f415daa64fae4161a34e676eeb66ee11" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39613 } } }
I20260926 07:02:01.769068 6021 sys_catalog.cc:458] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:01.769197 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:01.769668 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:01.771374 6025 catalog_manager.cc:1347] Generated new cluster ID: 05b24d402b1d4016aef4eae859888896
I20260926 07:02:01.771425 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:01.772423 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:01.761665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.1.217.126:39613)
0926 07:02:01.761826 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:01.761834 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:01.761858 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:02:01.762398 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:01.762405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:01.762450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:01.762568 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:01.763351 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.764360 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.765451 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.765853 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.765936 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:01.768628 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:01.768632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:01.768633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:01.768644 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:01.770708 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:01.771176 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:01.771178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:01.771179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:01.771286 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:01.771517 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:01.771518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:01.771519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:01.771825 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:01.771977 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:01.772092 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:02:01.772227 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260926 07:02:01.781071 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:01.781822 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:01.793303 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: Generated new TSK 0
I20260926 07:02:01.793526 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:02:01.824352 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:01.905411 6042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:01.905610 6042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:01.905634 6042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:01.906723 6042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:01.906759 6042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:01.906785 6042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:01.908198 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:01.908370 6042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:01.908571 6042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:01.913416 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:01.913420 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:01.913462 6042 server_base.cc:1034] running on GCE node
W20260926 07:02:01.913416 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:01.913851 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:01.914090 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:01.915227 6042 hybrid_clock.cc:648] HybridClock initialized: now 1790406121915217 us; error 28 us; skew 500 ppm
Sep 26 07:02:01 dist-test-slave-t1mc krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:01.917706 6042 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:02:01.918043 6042 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:01.918092 6042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:01.918255 6042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:01.919296 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "1ab83ac6ce1f66dd943a7432f8b68c4d"
server_key_iv: "5fd7ffe62eff05e385e20cd35c3c8918"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.919657 6042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "1ab83ac6ce1f66dd943a7432f8b68c4d"
server_key_iv: "5fd7ffe62eff05e385e20cd35c3c8918"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.920940 6042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:01.921605 6056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:01.921779 6042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:01.921876 6042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff"
format_stamp: "Formatted at 2026-09-26 07:02:01 on dist-test-slave-t1mc"
server_key: "1ab83ac6ce1f66dd943a7432f8b68c4d"
server_key_iv: "5fd7ffe62eff05e385e20cd35c3c8918"
server_key_version: "encryptionkey@0"
I20260926 07:02:01.921953 6042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:01.941119 6042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:01.942974 6042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:01.943141 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:01.943382 6042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:01.943701 6042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:01.943758 6042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:01.943797 6042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:01.943825 6042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:01.949620 6042 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:45259
I20260926 07:02:01.949662 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:45259 every 8 connection(s)
I20260926 07:02:01.949960 6042 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:02:01 dist-test-slave-t1mc krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406121, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:01.958853 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:01.950803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:39857 (local address 127.1.217.126:39613)
0926 07:02:01.950845 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:01.950847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:01.951060 (+ 213us) server_negotiation.cc:407] Connection header received
0926 07:02:01.951529 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:01.951530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:01.951539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:01.951591 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:01.952495 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.953056 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.953999 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.954192 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.954249 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:01.956233 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:01.956234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:01.956234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:01.956240 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:01.957742 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:01.958092 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:01.958092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:01.958093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:01.958142 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:01.958345 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:01.958346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:01.958346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:01.958441 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:01.958501 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:01.958732 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 07:02:01.958791 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:01.959048 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:01.950771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39613 (local address 127.1.217.65:39857)
0926 07:02:01.950929 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:01.950965 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:01.951437 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:01.951644 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:01.951647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:01.951890 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:01.952405 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:01.952411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.953132 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.953133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:01.953914 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:01.953916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:01.954261 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:01.954262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:01.954357 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:01.954794 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:01.954802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:01.956132 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:01.957810 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:01.957815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:01.957817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:01.958016 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:01.958201 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:01.958202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:01.958202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:01.958292 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:01.958506 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:01.958520 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:01.958625 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 07:02:01.958706 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:02:01.958817 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 07:02:01.959331 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6042
I20260926 07:02:01.959439 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:01.959555 6170 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39613
I20260926 07:02:01.959611 1893 external_mini_cluster.cc:1428] Setting key 309210ece4354cf7be105e18d29ca667
I20260926 07:02:01.959671 6170 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:01.959830 6170 heartbeater.cc:511] Master 127.1.217.126:39613 requested a full tablet report, sending...
I20260926 07:02:01.960384 5979 ts_manager.cc:194] Registered new tserver with Master: d619ed41d4cd4bcebcfb3f83c5ac05ff (127.1.217.65:45259)
I20260926 07:02:01.961076 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:39857
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:02:01.984583 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:02.066833 6175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.067080 6175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.067121 6175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.068280 6175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.068346 6175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.068390 6175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:02.070106 6175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.070367 6175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.070592 6175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:02.075508 6183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.075526 6180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.075519 6181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.075724 6175 server_base.cc:1034] running on GCE node
I20260926 07:02:02.075944 6175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.076211 6175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.077369 6175 hybrid_clock.cc:648] HybridClock initialized: now 1790406122077350 us; error 29 us; skew 500 ppm
Sep 26 07:02:02 dist-test-slave-t1mc krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:02.079941 6175 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:02:02.080284 6175 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.080358 6175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.080520 6175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.081667 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "3990f51315064e638a2c95103e5eb605"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "3d23419591fea089b829cde68526f54b"
server_key_iv: "2fbd11160f28d6e21d35bb3ad02851c5"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.082065 6175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "3990f51315064e638a2c95103e5eb605"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "3d23419591fea089b829cde68526f54b"
server_key_iv: "2fbd11160f28d6e21d35bb3ad02851c5"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.083333 6175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:02:02.084082 6189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.084254 6175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:02.084353 6175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "3990f51315064e638a2c95103e5eb605"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "3d23419591fea089b829cde68526f54b"
server_key_iv: "2fbd11160f28d6e21d35bb3ad02851c5"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.084434 6175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.094208 6175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.096001 6175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.096163 6175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.096371 6175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:02.096674 6175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:02.096730 6175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.096765 6175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:02.096839 6175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.102526 6175 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:37409
I20260926 07:02:02.102558 6302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:37409 every 8 connection(s)
I20260926 07:02:02.102816 6175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:02:02 dist-test-slave-t1mc krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:02.109205 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6175
I20260926 07:02:02.109328 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:02.109542 1893 external_mini_cluster.cc:1428] Setting key 17096bbfbbd48aa39203e7ccaf0cdf61
I20260926 07:02:02.112093 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.103529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:34931 (local address 127.1.217.126:39613)
0926 07:02:02.103572 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.103574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.103769 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:02:02.104218 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.104219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.104228 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.104284 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.105261 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.105738 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.106695 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.106874 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.106924 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.108924 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.108926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.108926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:02.108932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.110738 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.111110 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.111111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.111111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.111172 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.111391 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.111392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.111392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.111547 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.111613 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.111875 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.111982 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:02.112295 6305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.103494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39613 (local address 127.1.217.66:34931)
0926 07:02:02.103639 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.103679 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.104138 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.104335 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.104338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.104599 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:02.105174 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.105180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.105814 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.105814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.106618 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.106620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.106942 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.106943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.107039 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.107448 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:02.107456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:02.108846 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:02.110815 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:02.110820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:02.110822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:02.111038 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:02.111236 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:02.111236 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:02.111237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:02.111331 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:02.111607 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:02.111611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:02.111712 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.111846 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.112016 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 07:02:02.112748 6303 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39613
I20260926 07:02:02.112881 6303 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:02.113023 6303 heartbeater.cc:511] Master 127.1.217.126:39613 requested a full tablet report, sending...
I20260926 07:02:02.113368 5979 ts_manager.cc:194] Registered new tserver with Master: 3990f51315064e638a2c95103e5eb605 (127.1.217.66:37409)
I20260926 07:02:02.113750 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:34931
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:02:02.135574 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:02.217896 6308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.218122 6308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.218161 6308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.219295 6308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.219358 6308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.219403 6308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:02.220965 6308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.221225 6308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.221464 6308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:02.226454 6313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.226420 6316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.226572 6308 server_base.cc:1034] running on GCE node
W20260926 07:02:02.226460 6314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.226840 6308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.227088 6308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.228229 6308 hybrid_clock.cc:648] HybridClock initialized: now 1790406122228222 us; error 32 us; skew 500 ppm
Sep 26 07:02:02 dist-test-slave-t1mc krb5kdc[5936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:02.230652 6308 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:02:02.231019 6308 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.231067 6308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.231153 6308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.232110 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "b91063c81157424c9a691d8d75a1bb4e"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "a6b4a2226319177cc59573da779be0b1"
server_key_iv: "39e9f8e1ce405c96355c6c2382ab1cdb"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.232398 6308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "b91063c81157424c9a691d8d75a1bb4e"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "a6b4a2226319177cc59573da779be0b1"
server_key_iv: "39e9f8e1ce405c96355c6c2382ab1cdb"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.233654 6308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:02:02.234293 6322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.234495 6308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:02.234589 6308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "b91063c81157424c9a691d8d75a1bb4e"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "a6b4a2226319177cc59573da779be0b1"
server_key_iv: "39e9f8e1ce405c96355c6c2382ab1cdb"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.234669 6308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.251892 6308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.254040 6308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.254197 6308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.254424 6308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:02.254745 6308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:02.254799 6308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.254841 6308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:02.254869 6308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.260747 6308 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:33531
I20260926 07:02:02.260804 6435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:33531 every 8 connection(s)
I20260926 07:02:02.261067 6308 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:02:02 dist-test-slave-t1mc krb5kdc[5936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406122, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:02.270159 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.261780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60477 (local address 127.1.217.126:39613)
0926 07:02:02.261842 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.261844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.262064 (+ 220us) server_negotiation.cc:407] Connection header received
0926 07:02:02.262505 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.262506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.262516 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.262570 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.263477 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.263958 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.265095 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.265281 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.265334 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.267368 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.267370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.267370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:02.267378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.268674 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.269173 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.269174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.269175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.269236 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.269493 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.269494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.269494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.269605 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.269679 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.269956 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.270063 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:02:02.270291 6438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.261747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39613 (local address 127.1.217.67:60477)
0926 07:02:02.261916 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.261957 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.262425 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.262628 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.262632 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.262882 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:02.263399 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.263404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.264056 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.264057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.265023 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.265025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.265351 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.265352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.265443 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.265949 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:02.265958 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:02.267273 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:02.268758 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:02.268764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:02.268793 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:02.269086 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:02.269303 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:02.269304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:02.269305 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:02.269420 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:02.269669 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:02.269675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:02.269819 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.269926 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.270076 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260926 07:02:02.270709 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6308
I20260926 07:02:02.270812 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:02.270813 6436 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39613
I20260926 07:02:02.270915 6436 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:02.271014 1893 external_mini_cluster.cc:1428] Setting key 8c9e880849333d56efbf59f05db1ca9b
I20260926 07:02:02.271083 6436 heartbeater.cc:511] Master 127.1.217.126:39613 requested a full tablet report, sending...
I20260926 07:02:02.271489 5979 ts_manager.cc:194] Registered new tserver with Master: b91063c81157424c9a691d8d75a1bb4e (127.1.217.67:33531)
I20260926 07:02:02.271924 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:02:02.271973 5979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:60477
I20260926 07:02:02.278839 6019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.273320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35078 (local address 127.1.217.126:39613)
0926 07:02:02.273386 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.273388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.273434 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:02:02.273492 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.273492 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.273500 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.273538 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.274019 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.274471 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.275165 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.275341 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.275395 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.276491 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.276492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.276492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:02.276498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.277900 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.278198 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.278198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.278199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.278247 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:02.278399 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:02.278400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:02.278400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:02.278528 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.278605 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.278680 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.278732 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:02:02.280234 5979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:02.281840 5979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:02.287837 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.283894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:33531 (local address 127.0.0.1:55004)
0926 07:02:02.284184 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.284193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.284241 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.284589 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.284593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.284595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:02.284661 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.284663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.285642 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.285642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.286660 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.286662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.287548 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.287549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.287626 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.287640 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.287714 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.287757 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":59,"threads_started":1}
I20260926 07:02:02.287860 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.284231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.1.217.67:33531)
0926 07:02:02.284400 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.284404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.284414 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:02.284455 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.284457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.284470 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.284566 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:02.284754 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.285547 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.286764 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.287466 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.287537 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.287590 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.287739 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.287780 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":75,"threads_started":1}
I20260926 07:02:02.288542 6370 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 (DEFAULT_TABLE table=test-table [id=f72be87517cd488c93965a08d3154157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:02.288898 6370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c282d7f114243ad820fe63ea8c487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:02.289528 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.284084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:45259 (local address 127.0.0.1:39526)
0926 07:02:02.284638 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.284648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.284690 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.284866 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.284867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.284869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:02.284902 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.284904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.287207 (+ 2303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.287208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.288372 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.288374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.289297 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.289298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.289358 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.289362 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.289419 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.289461 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":55,"threads_started":1}
I20260926 07:02:02.289561 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.284220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39526 (local address 127.1.217.65:45259)
0926 07:02:02.284703 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.284708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.284717 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:02.284755 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.284757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.284794 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.284870 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:02.284976 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.286051 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.288441 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.289206 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.289295 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.289350 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.289433 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.289476 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":108,"threads_started":1}
I20260926 07:02:02.290273 6104 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 (DEFAULT_TABLE table=test-table [id=f72be87517cd488c93965a08d3154157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:02.290575 6104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c282d7f114243ad820fe63ea8c487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:02.291703 6450 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Bootstrap starting.
I20260926 07:02:02.292162 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.283875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:37409 (local address 127.0.0.1:49852)
0926 07:02:02.284001 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.284039 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.284103 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.284830 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.284831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.284833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:02.284892 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.284894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.286164 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.286164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.287070 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.287072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.291902 (+ 4830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.291903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.291993 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.291998 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.292061 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.292099 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260926 07:02:02.292223 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.284080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.1.217.66:37409)
0926 07:02:02.284400 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.284404 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.284412 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:02.284448 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.284449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.284461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.284529 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:02.284997 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.286060 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.287169 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.287897 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.288211 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.288272 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.292094 (+ 3822us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.292146 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":70,"threads_started":1}
I20260926 07:02:02.292330 6450 tablet_bootstrap.cc:654] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:02.292974 6450 log.cc:824] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:02.292937 6237 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 (DEFAULT_TABLE table=test-table [id=f72be87517cd488c93965a08d3154157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:02.293254 6237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1c282d7f114243ad820fe63ea8c487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:02.293565 6452 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Bootstrap starting.
I20260926 07:02:02.293792 6450 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: No bootstrap required, opened a new log
I20260926 07:02:02.293892 6450 ts_tablet_manager.cc:1397] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:02:02.294107 6452 tablet_bootstrap.cc:654] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:02.294539 6452 log.cc:824] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:02.295184 6452 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: No bootstrap required, opened a new log
I20260926 07:02:02.295256 6452 ts_tablet_manager.cc:1397] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:02.296228 6450 raft_consensus.cc:348] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.296438 6454 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Bootstrap starting.
I20260926 07:02:02.296484 6450 raft_consensus.cc:374] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:02.296586 6450 raft_consensus.cc:729] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91063c81157424c9a691d8d75a1bb4e, State: Initialized, Role: FOLLOWER
I20260926 07:02:02.296710 6450 consensus_queue.cc:260] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.296859 6452 raft_consensus.cc:348] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.296999 6450 ts_tablet_manager.cc:1428] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:02:02.297039 6452 raft_consensus.cc:374] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:02.297055 6454 tablet_bootstrap.cc:654] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:02.297088 6452 raft_consensus.cc:729] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d619ed41d4cd4bcebcfb3f83c5ac05ff, State: Initialized, Role: FOLLOWER
I20260926 07:02:02.297125 6436 heartbeater.cc:503] Master 127.1.217.126:39613 was elected leader, sending a full tablet report...
I20260926 07:02:02.297230 6452 consensus_queue.cc:260] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.297528 6452 ts_tablet_manager.cc:1428] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:02:02.297631 6454 log.cc:824] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:02.297677 6170 heartbeater.cc:503] Master 127.1.217.126:39613 was elected leader, sending a full tablet report...
I20260926 07:02:02.298346 6454 tablet_bootstrap.cc:492] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: No bootstrap required, opened a new log
I20260926 07:02:02.298415 6454 ts_tablet_manager.cc:1397] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:02.299691 6454 raft_consensus.cc:348] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.299840 6454 raft_consensus.cc:374] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:02.299885 6454 raft_consensus.cc:729] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3990f51315064e638a2c95103e5eb605, State: Initialized, Role: FOLLOWER
I20260926 07:02:02.299985 6454 consensus_queue.cc:260] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.300222 6454 ts_tablet_manager.cc:1428] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:02:02.300264 6303 heartbeater.cc:503] Master 127.1.217.126:39613 was elected leader, sending a full tablet report...
W20260926 07:02:02.353353 6304 tablet.cc:2369] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:02.445399 6456 raft_consensus.cc:482] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:02.445529 6456 raft_consensus.cc:504] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.445966 6456 leader_election.cc:288] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3990f51315064e638a2c95103e5eb605 (127.1.217.66:37409), b91063c81157424c9a691d8d75a1bb4e (127.1.217.67:33531)
I20260926 07:02:02.449891 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.446243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:45579 (local address 127.1.217.66:37409)
0926 07:02:02.446292 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.446294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.446321 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:02:02.446351 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.446352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.446375 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.446444 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:02.446619 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.447381 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.448941 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.449532 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.449610 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.449618 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.449780 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.449818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:02:02.449891 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.446335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:53467 (local address 127.1.217.67:33531)
0926 07:02:02.446416 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.446418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.446426 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:02.446450 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.446451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.446459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.446519 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:02.446665 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.447408 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.448944 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.449531 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.449610 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.449618 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.449780 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.449818 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 07:02:02.449914 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.446170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:37409 (local address 127.1.217.65:45579)
0926 07:02:02.446231 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.446243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.446286 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.446450 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.446450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.446452 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:02.446539 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.446541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.447497 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.447499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.448846 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.448849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.449621 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.449622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.449699 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.449701 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.449759 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.449816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":102}
I20260926 07:02:02.449920 6459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.446171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:33531 (local address 127.1.217.65:53467)
0926 07:02:02.446293 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.446305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.446364 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.446531 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.446532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.446533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:02.446591 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.446592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.447497 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.447499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.448845 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:02.448849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.449622 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.449622 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:02.449697 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.449701 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.449759 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.449816 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":29,"thread_start_us":58,"threads_started":1}
I20260926 07:02:02.450311 6390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c282d7f114243ad820fe63ea8c487" candidate_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91063c81157424c9a691d8d75a1bb4e" is_pre_election: true
I20260926 07:02:02.450495 6390 raft_consensus.cc:2454] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d619ed41d4cd4bcebcfb3f83c5ac05ff in term 0.
I20260926 07:02:02.450493 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c282d7f114243ad820fe63ea8c487" candidate_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3990f51315064e638a2c95103e5eb605" is_pre_election: true
I20260926 07:02:02.450661 6257 raft_consensus.cc:2454] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d619ed41d4cd4bcebcfb3f83c5ac05ff in term 0.
W20260926 07:02:02.450686 6171 tablet.cc:2369] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:02.450918 6058 leader_election.cc:302] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b91063c81157424c9a691d8d75a1bb4e, d619ed41d4cd4bcebcfb3f83c5ac05ff; no voters:
I20260926 07:02:02.451108 6456 raft_consensus.cc:2781] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:02.451165 6456 raft_consensus.cc:482] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:02.451188 6456 raft_consensus.cc:3037] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:02.451829 6456 raft_consensus.cc:504] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.451977 6456 leader_election.cc:288] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [CANDIDATE]: Term 1 election: Requested vote from peers 3990f51315064e638a2c95103e5eb605 (127.1.217.66:37409), b91063c81157424c9a691d8d75a1bb4e (127.1.217.67:33531)
I20260926 07:02:02.452131 6390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c282d7f114243ad820fe63ea8c487" candidate_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91063c81157424c9a691d8d75a1bb4e"
I20260926 07:02:02.452132 6257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1c282d7f114243ad820fe63ea8c487" candidate_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3990f51315064e638a2c95103e5eb605"
I20260926 07:02:02.452209 6257 raft_consensus.cc:3037] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:02.452211 6390 raft_consensus.cc:3037] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:02.452967 6257 raft_consensus.cc:2454] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d619ed41d4cd4bcebcfb3f83c5ac05ff in term 1.
I20260926 07:02:02.453018 6390 raft_consensus.cc:2454] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d619ed41d4cd4bcebcfb3f83c5ac05ff in term 1.
I20260926 07:02:02.453186 6059 leader_election.cc:302] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3990f51315064e638a2c95103e5eb605, d619ed41d4cd4bcebcfb3f83c5ac05ff; no voters:
I20260926 07:02:02.453289 6456 raft_consensus.cc:2781] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:02.453444 6456 raft_consensus.cc:686] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [term 1 LEADER]: Becoming Leader. State: Replica: d619ed41d4cd4bcebcfb3f83c5ac05ff, State: Running, Role: LEADER
I20260926 07:02:02.453537 6456 consensus_queue.cc:237] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } }
I20260926 07:02:02.454387 5980 catalog_manager.cc:5495] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff reported cstate change: term changed from 0 to 1, leader changed from <none> to d619ed41d4cd4bcebcfb3f83c5ac05ff (127.1.217.65). New cstate: current_term: 1 leader_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d619ed41d4cd4bcebcfb3f83c5ac05ff" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 45259 } health_report { overall_health: HEALTHY } } }
I20260926 07:02:02.467175 6449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.464953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.1.217.65:45259)
0926 07:02:02.465022 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.465025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.465033 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:02.465111 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.465112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.465128 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.465192 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:02.465298 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.465743 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.466507 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:02.466678 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:02.466726 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:02.466861 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:02.466949 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:02.467012 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.467040 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.467055 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:02:02.468902 6390 raft_consensus.cc:1261] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 1 FOLLOWER]: Refusing update from remote peer d619ed41d4cd4bcebcfb3f83c5ac05ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:02.468900 6257 raft_consensus.cc:1261] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 1 FOLLOWER]: Refusing update from remote peer d619ed41d4cd4bcebcfb3f83c5ac05ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:02.469274 6460 consensus_queue.cc:1035] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "3990f51315064e638a2c95103e5eb605" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:02.469411 6456 consensus_queue.cc:1035] T 2b1c282d7f114243ad820fe63ea8c487 P d619ed41d4cd4bcebcfb3f83c5ac05ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91063c81157424c9a691d8d75a1bb4e" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:02.472112 6464 mvcc.cc:205] Tried to move back new op lower bound from 7333503477629882368 to 7333503477570334720. Current Snapshot: MvccSnapshot[applied={T|T < 7333503477629882368}]
I20260926 07:02:02.472609 6465 mvcc.cc:205] Tried to move back new op lower bound from 7333503477629882368 to 7333503477570334720. Current Snapshot: MvccSnapshot[applied={T|T < 7333503477629882368}]
I20260926 07:02:02.472802 6463 mvcc.cc:205] Tried to move back new op lower bound from 7333503477629882368 to 7333503477570334720. Current Snapshot: MvccSnapshot[applied={T|T < 7333503477629882368 or (T in {7333503477629882368})}]
I20260926 07:02:02.474198 5978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:02.474291 5978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:02.475004 5978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f415daa64fae4161a34e676eeb66ee11: Sending DeleteTablet for 3 replicas of tablet 2b1c282d7f114243ad820fe63ea8c487
I20260926 07:02:02.475353 6370 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:02 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:55004
I20260926 07:02:02.475404 6104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:02 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:39526
I20260926 07:02:02.475634 6473 tablet_replica.cc:330] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: stopping tablet replica
I20260926 07:02:02.475617 6237 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b1c282d7f114243ad820fe63ea8c487 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:02 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:49852
I20260926 07:02:02.475797 6475 tablet_replica.cc:330] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: stopping tablet replica
I20260926 07:02:02.475855 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6042
I20260926 07:02:02.475881 6475 raft_consensus.cc:2229] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:02.475972 6475 raft_consensus.cc:2258] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:02.476627 6475 ts_tablet_manager.cc:1905] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:02.478121 6473 raft_consensus.cc:2229] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:02.478243 6473 raft_consensus.cc:2258] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:02.478458 6475 ts_tablet_manager.cc:1918] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:02.478531 6475 log.cc:1196] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/2b1c282d7f114243ad820fe63ea8c487
I20260926 07:02:02.478675 6473 ts_tablet_manager.cc:1905] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:02.478845 6475 ts_tablet_manager.cc:1939] T 2b1c282d7f114243ad820fe63ea8c487 P 3990f51315064e638a2c95103e5eb605: Deleting consensus metadata
I20260926 07:02:02.479240 5966 catalog_manager.cc:4841] TS 3990f51315064e638a2c95103e5eb605 (127.1.217.66:37409): tablet 2b1c282d7f114243ad820fe63ea8c487 (table test-table [id=f72be87517cd488c93965a08d3154157]) successfully deleted
I20260926 07:02:02.480533 6473 ts_tablet_manager.cc:1918] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:02.480609 6473 log.cc:1196] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/2b1c282d7f114243ad820fe63ea8c487
I20260926 07:02:02.480901 6473 ts_tablet_manager.cc:1939] T 2b1c282d7f114243ad820fe63ea8c487 P b91063c81157424c9a691d8d75a1bb4e: Deleting consensus metadata
I20260926 07:02:02.481328 5965 catalog_manager.cc:4841] TS b91063c81157424c9a691d8d75a1bb4e (127.1.217.67:33531): tablet 2b1c282d7f114243ad820fe63ea8c487 (table test-table [id=f72be87517cd488c93965a08d3154157]) successfully deleted
W20260926 07:02:02.482167 5965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:45259 (error 108)
I20260926 07:02:02.482375 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6175
I20260926 07:02:02.482640 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.482520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:45259 (local address 127.0.0.1:39536)
0926 07:02:02.482570 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.482601 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:45259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:02:02.483052 5965 catalog_manager.cc:4584] TS d619ed41d4cd4bcebcfb3f83c5ac05ff (127.1.217.65:45259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b1c282d7f114243ad820fe63ea8c487: Network error: Client connection negotiation failed: client connection to 127.1.217.65:45259: connect: Connection refused (error 111)
I20260926 07:02:02.487857 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6308
I20260926 07:02:02.492138 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 5949
2026-09-26T07:02:02Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2017 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T07:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:02Z Disabled control of system clock
I20260926 07:02:02.518769 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:41537
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:41537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:02:02.597141 6480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.597343 6480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.597363 6480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.598413 6480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:02.598446 6480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.598459 6480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.598471 6480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:02.598484 6480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:02.599709 6480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:41537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:41537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.599890 6480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.600107 6480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:02:02.604905 6480 server_base.cc:1034] running on GCE node
W20260926 07:02:02.604882 6486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.604852 6485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.604862 6488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.605311 6480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.605633 6480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.606788 6480 hybrid_clock.cc:648] HybridClock initialized: now 1790406122606783 us; error 36 us; skew 500 ppm
I20260926 07:02:02.607923 6480 webserver.cc:460] Webserver started at http://127.1.217.126:37891/ using document root <none> and password file <none>
I20260926 07:02:02.608155 6480 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.608228 6480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.608361 6480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.609448 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "8ee5c2597ce94075905264bddc6ffacb"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "9090073239b890c7eb895344250392c8"
server_key_iv: "34cf97e8e3f2949bdc3c06d8c2ad2d5c"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.609812 6480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "8ee5c2597ce94075905264bddc6ffacb"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "9090073239b890c7eb895344250392c8"
server_key_iv: "34cf97e8e3f2949bdc3c06d8c2ad2d5c"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.611089 6480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:02.611827 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.612007 6480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:02.612103 6480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "8ee5c2597ce94075905264bddc6ffacb"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "9090073239b890c7eb895344250392c8"
server_key_iv: "34cf97e8e3f2949bdc3c06d8c2ad2d5c"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.612185 6480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.615370 6480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.615628 6480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.615773 6480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.619423 6480 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:41537
I20260926 07:02:02.619465 6546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:41537 every 8 connection(s)
I20260926 07:02:02.619781 6480 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:02.620411 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:02.622509 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6480
I20260926 07:02:02.622615 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:02.622795 1893 external_mini_cluster.cc:1428] Setting key baba2d181392baedc1a3796e0f29b8e2
I20260926 07:02:02.623389 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: Bootstrap starting.
I20260926 07:02:02.623961 6547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:02.624254 6547 log.cc:824] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:02.625074 6547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: No bootstrap required, opened a new log
I20260926 07:02:02.625273 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.623296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.1.217.126:41537)
0926 07:02:02.623496 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.623505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.623540 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:02:02.623918 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.623924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.623960 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.624136 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.624812 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.624816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.624816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:02.624820 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.624868 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:02.624878 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.624989 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.625025 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.625080 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260926 07:02:02.626811 6547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } }
I20260926 07:02:02.626938 6547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:02.626959 6547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee5c2597ce94075905264bddc6ffacb, State: Initialized, Role: FOLLOWER
I20260926 07:02:02.627097 6547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } }
I20260926 07:02:02.627427 6547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:02.627471 6547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:02.627519 6547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:02.628039 6547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } }
I20260926 07:02:02.628145 6547 leader_election.cc:302] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee5c2597ce94075905264bddc6ffacb; no voters:
I20260926 07:02:02.628320 6547 leader_election.cc:288] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:02.628381 6552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:02.628517 6552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [term 1 LEADER]: Becoming Leader. State: Replica: 8ee5c2597ce94075905264bddc6ffacb, State: Running, Role: LEADER
I20260926 07:02:02.628571 6547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:02.628609 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } }
I20260926 07:02:02.628996 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee5c2597ce94075905264bddc6ffacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } } }
I20260926 07:02:02.629130 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:02.629005 6554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee5c2597ce94075905264bddc6ffacb. Latest consensus state: current_term: 1 leader_uuid: "8ee5c2597ce94075905264bddc6ffacb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee5c2597ce94075905264bddc6ffacb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 41537 } } }
I20260926 07:02:02.629391 6554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb [sys.catalog]: This master's current role is: LEADER
W20260926 07:02:02.630326 6568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:02:02.630388 6568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:02:02.630456 6561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:02.630616 6561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:02.631948 6561 catalog_manager.cc:1347] Generated new cluster ID: 10bff786e8be4e488b2249ab3fe856a1
I20260926 07:02:02.632002 6561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:02.640026 6561 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:02.640197 6561 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:02.643678 6561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: Generated new TSK 0
I20260926 07:02:02.643853 6561 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:02.647817 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--builtin_ntp_servers=127.1.217.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:02:02.726934 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.727136 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.727160 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.728333 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.728374 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.728400 6571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:02.729858 6571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.730127 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.730413 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:02.736301 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.736301 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.736462 6571 server_base.cc:1034] running on GCE node
W20260926 07:02:02.736552 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.736840 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.737128 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.738284 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790406122738274 us; error 27 us; skew 500 ppm
I20260926 07:02:02.739543 6571 webserver.cc:460] Webserver started at http://127.1.217.65:41329/ using document root <none> and password file <none>
I20260926 07:02:02.739809 6571 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.739854 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.739989 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.741112 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "65a6ab073f304a28a8fa36f7fee2b8d3"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30357121a1d29c0afbffd46947729e33"
server_key_iv: "50b3dc0b36d7914317762c6cf1ef1dc9"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.741457 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "65a6ab073f304a28a8fa36f7fee2b8d3"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30357121a1d29c0afbffd46947729e33"
server_key_iv: "50b3dc0b36d7914317762c6cf1ef1dc9"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.742823 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:02.743587 6585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.743739 6571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:02.743835 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "65a6ab073f304a28a8fa36f7fee2b8d3"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30357121a1d29c0afbffd46947729e33"
server_key_iv: "50b3dc0b36d7914317762c6cf1ef1dc9"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.743908 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.747202 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.747467 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.747596 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.747838 6571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:02.748201 6571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:02.748238 6571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.748258 6571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:02.748302 6571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.755316 6571 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:34563
I20260926 07:02:02.755380 6698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:34563 every 8 connection(s)
I20260926 07:02:02.755692 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:02.758677 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.756410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:40575 (local address 127.1.217.126:41537)
0926 07:02:02.756463 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.756466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.756720 (+ 254us) server_negotiation.cc:407] Connection header received
0926 07:02:02.757214 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.757216 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.757220 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.757272 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.758325 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.758326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.758326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:02.758331 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.758346 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:02.758351 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.758416 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.758508 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.758524 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:02:02.758817 6701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.756385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:41537 (local address 127.1.217.65:40575)
0926 07:02:02.756590 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.756633 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.757127 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.757322 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.757326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.757334 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:02.758166 (+ 832us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:02.758167 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:02.758192 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:02.758243 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:02.758415 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:02.758420 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:02.758423 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.758515 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.758596 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260926 07:02:02.759470 6699 heartbeater.cc:347] Connected to a master server at 127.1.217.126:41537
I20260926 07:02:02.759544 6699 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:02.759706 6699 heartbeater.cc:511] Master 127.1.217.126:41537 requested a full tablet report, sending...
I20260926 07:02:02.760180 6511 ts_manager.cc:194] Registered new tserver with Master: 65a6ab073f304a28a8fa36f7fee2b8d3 (127.1.217.65:34563)
I20260926 07:02:02.762478 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6571
I20260926 07:02:02.762580 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:02.762753 1893 external_mini_cluster.cc:1428] Setting key 1a1f5b0b8bf8b620d1d5fe436d58b419
I20260926 07:02:02.763612 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--builtin_ntp_servers=127.1.217.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:02:02.842085 6702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.842309 6702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.842345 6702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.843472 6702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.843533 6702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.843575 6702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:02.844918 6702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.845176 6702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.845454 6702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:02.850282 6708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.850277 6710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.850329 6707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.850450 6702 server_base.cc:1034] running on GCE node
I20260926 07:02:02.850800 6702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.851061 6702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.852217 6702 hybrid_clock.cc:648] HybridClock initialized: now 1790406122852199 us; error 35 us; skew 500 ppm
I20260926 07:02:02.853448 6702 webserver.cc:460] Webserver started at http://127.1.217.66:34037/ using document root <none> and password file <none>
I20260926 07:02:02.853691 6702 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.853761 6702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.853909 6702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.854948 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "e3677dc779234f49ab8789cea72a5dd5"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "e885895ef05d835bec2f576325614301"
server_key_iv: "469c0af8fabf71033e0f0c8ecd6bb074"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.855307 6702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "e3677dc779234f49ab8789cea72a5dd5"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "e885895ef05d835bec2f576325614301"
server_key_iv: "469c0af8fabf71033e0f0c8ecd6bb074"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.856527 6702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:02:02.857353 6716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.857554 6702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.857631 6702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "e3677dc779234f49ab8789cea72a5dd5"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "e885895ef05d835bec2f576325614301"
server_key_iv: "469c0af8fabf71033e0f0c8ecd6bb074"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.857693 6702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.861397 6702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.861652 6702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.861768 6702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.861924 6702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:02.862165 6702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:02.862193 6702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.862212 6702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:02.862226 6702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.867827 6702 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:46239
I20260926 07:02:02.867867 6829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:46239 every 8 connection(s)
I20260926 07:02:02.868163 6702 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:02.871060 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.868885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:57195 (local address 127.1.217.126:41537)
0926 07:02:02.868948 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.868950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.869173 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:02:02.869671 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.869672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.869678 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.869734 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.870819 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.870820 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.870821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:02.870826 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.870840 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:02.870844 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.870895 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.870977 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.870992 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:02:02.871232 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.868852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:41537 (local address 127.1.217.66:57195)
0926 07:02:02.869061 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.869092 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.869571 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.869801 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.869805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.869814 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:02.870676 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:02.870676 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:02.870702 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:02.870728 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:02.870893 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:02.870897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:02.870901 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.870969 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.871028 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260926 07:02:02.871719 6830 heartbeater.cc:347] Connected to a master server at 127.1.217.126:41537
I20260926 07:02:02.871791 6830 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:02.871941 6830 heartbeater.cc:511] Master 127.1.217.126:41537 requested a full tablet report, sending...
I20260926 07:02:02.872331 6511 ts_manager.cc:194] Registered new tserver with Master: e3677dc779234f49ab8789cea72a5dd5 (127.1.217.66:46239)
I20260926 07:02:02.878180 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6702
I20260926 07:02:02.878254 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:02.878389 1893 external_mini_cluster.cc:1428] Setting key c2afa374da77a971c6057d490f4b692b
I20260926 07:02:02.879184 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--builtin_ntp_servers=127.1.217.84:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 07:02:02.959517 6833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:02.959707 6833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:02.959728 6833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:02.960922 6833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:02.960961 6833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:02.960987 6833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:02.962333 6833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:02.962515 6833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:02.962745 6833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:02:02.967729 6833 server_base.cc:1034] running on GCE node
W20260926 07:02:02.967685 6839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.967703 6838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:02.967686 6841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:02.968102 6833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:02.968384 6833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:02.969540 6833 hybrid_clock.cc:648] HybridClock initialized: now 1790406122969526 us; error 43 us; skew 500 ppm
I20260926 07:02:02.970786 6833 webserver.cc:460] Webserver started at http://127.1.217.67:33627/ using document root <none> and password file <none>
I20260926 07:02:02.971040 6833 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:02.971122 6833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:02.971271 6833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:02.972322 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "59d935e923dd408592a4313a8e478aba"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30680da4c9da4ff080b5dd0ff77ad905"
server_key_iv: "3f381b656b38f02360aedd6837db0fce"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.972698 6833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "59d935e923dd408592a4313a8e478aba"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30680da4c9da4ff080b5dd0ff77ad905"
server_key_iv: "3f381b656b38f02360aedd6837db0fce"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.974013 6833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:02.974689 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.974862 6833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:02.974958 6833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "59d935e923dd408592a4313a8e478aba"
format_stamp: "Formatted at 2026-09-26 07:02:02 on dist-test-slave-t1mc"
server_key: "30680da4c9da4ff080b5dd0ff77ad905"
server_key_iv: "3f381b656b38f02360aedd6837db0fce"
server_key_version: "encryptionkey@0"
I20260926 07:02:02.975037 6833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:02.978421 6833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:02.978683 6833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:02.978820 6833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:02.979036 6833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:02.979331 6833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:02.979386 6833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.979424 6833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:02.979452 6833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:02.985170 6833 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:46553
I20260926 07:02:02.985204 6960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:46553 every 8 connection(s)
I20260926 07:02:02.985519 6833 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:02.988282 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.986256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:53669 (local address 127.1.217.126:41537)
0926 07:02:02.986303 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.986305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.986560 (+ 255us) server_negotiation.cc:407] Connection header received
0926 07:02:02.986991 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.986992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.986996 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.987057 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.988095 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.988096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.988096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:02.988101 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.988115 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:02.988119 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.988170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.988219 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.988231 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:02.988468 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.986254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:41537 (local address 127.1.217.67:53669)
0926 07:02:02.986433 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:02.986464 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:02.986895 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:02.987109 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:02.987112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:02.987120 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:02.987960 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:02.987961 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:02.987987 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:02.988013 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:02.988171 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:02.988173 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:02.988176 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:02.988224 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:02:02.988267 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260926 07:02:02.988972 6961 heartbeater.cc:347] Connected to a master server at 127.1.217.126:41537
I20260926 07:02:02.989058 6961 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:02.989163 6961 heartbeater.cc:511] Master 127.1.217.126:41537 requested a full tablet report, sending...
I20260926 07:02:02.989457 6511 ts_manager.cc:194] Registered new tserver with Master: 59d935e923dd408592a4313a8e478aba (127.1.217.67:46553)
I20260926 07:02:02.993687 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 6833
I20260926 07:02:02.993768 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:02.993924 1893 external_mini_cluster.cc:1428] Setting key 1a42278ee3f065daaa9ff725dd50f32f
I20260926 07:02:02.994678 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:02:02.996848 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:02.995827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.1.217.126:41537)
0926 07:02:02.995863 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:02.995865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:02.995982 (+ 117us) server_negotiation.cc:407] Connection header received
0926 07:02:02.996049 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:02.996049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:02.996053 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:02.996093 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:02.996625 (+ 532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:02.996626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:02.996626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:02.996629 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:02.996642 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:02.996645 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:02.996693 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:02.996726 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 07:02:02.996737 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:02.998520 6511 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:02.999954 6511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:03.003682 6969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.001931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34563 (local address 127.0.0.1:39764)
0926 07:02:03.002108 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.002146 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.002200 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.002730 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.002731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.002733 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:03.003238 (+ 505us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:03.003238 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:03.003254 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:03.003277 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:03.003528 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:03.003535 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:03.003538 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:03.003604 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 07:02:03.003618 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260926 07:02:03.003873 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.002357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.1.217.67:46553)
0926 07:02:03.002530 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.002535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.002543 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:03.002564 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.002566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.002573 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.002655 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.003580 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.003585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.003586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.003590 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.003618 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.003637 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.003716 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.003779 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.003792 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260926 07:02:03.004071 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.002232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.1.217.66:46239)
0926 07:02:03.002946 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.002951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.002960 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:03.002984 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.002986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.002995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.003072 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.003597 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.003604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.003604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.003609 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.003641 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.003667 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.003894 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.003917 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.003985 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":78,"threads_started":1}
I20260926 07:02:03.004170 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.002062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.1.217.65:34563)
0926 07:02:03.002343 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.002349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.002358 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:03.002385 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.002388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.002398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.002474 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.003363 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.003373 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.003374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.003379 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.003424 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.003453 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.003522 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.004065 (+ 543us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.004082 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260926 07:02:03.003858 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.001983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46239 (local address 127.0.0.1:60300)
0926 07:02:03.002111 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.002146 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.002200 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.003085 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.003085 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.003087 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:03.003470 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:03.003470 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:03.003476 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:03.003485 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:03.003752 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:03.003753 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:03.003754 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 07:02:03.003786 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 07:02:03.003799 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260926 07:02:03.003857 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.002169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:46553 (local address 127.0.0.1:37354)
0926 07:02:03.002400 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.002409 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.002466 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.002671 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.002676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.002681 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:03.003504 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:03.003505 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:03.003512 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:03.003522 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:03.003711 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:03.003712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:03.003712 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 07:02:03.003770 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 07:02:03.003783 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":55,"threads_started":1}
I20260926 07:02:03.004962 6633 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 (DEFAULT_TABLE table=test-table [id=7112050584884dee802625d826cde2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:03.005115 6895 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 (DEFAULT_TABLE table=test-table [id=7112050584884dee802625d826cde2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:03.005115 6764 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 (DEFAULT_TABLE table=test-table [id=7112050584884dee802625d826cde2ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:03.005324 6633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7418cc6a4942638e8d674e13f9ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.005436 6895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7418cc6a4942638e8d674e13f9ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.005436 6764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7418cc6a4942638e8d674e13f9ee43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.007864 6975 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: Bootstrap starting.
I20260926 07:02:03.008533 6975 tablet_bootstrap.cc:654] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.008569 6976 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Bootstrap starting.
I20260926 07:02:03.008891 6977 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Bootstrap starting.
I20260926 07:02:03.009006 6975 log.cc:824] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.009246 6976 tablet_bootstrap.cc:654] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.009562 6977 tablet_bootstrap.cc:654] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.009575 6976 log.cc:824] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.009804 6975 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: No bootstrap required, opened a new log
I20260926 07:02:03.009868 6975 ts_tablet_manager.cc:1397] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:03.009907 6977 log.cc:824] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.010310 6976 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: No bootstrap required, opened a new log
I20260926 07:02:03.010412 6976 ts_tablet_manager.cc:1397] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:02:03.010602 6977 tablet_bootstrap.cc:492] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: No bootstrap required, opened a new log
I20260926 07:02:03.010676 6977 ts_tablet_manager.cc:1397] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:03.012476 6975 raft_consensus.cc:348] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.012674 6975 raft_consensus.cc:374] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.012714 6975 raft_consensus.cc:729] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a6ab073f304a28a8fa36f7fee2b8d3, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.012717 6977 raft_consensus.cc:348] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.012859 6975 consensus_queue.cc:260] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.012944 6977 raft_consensus.cc:374] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.012989 6977 raft_consensus.cc:729] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3677dc779234f49ab8789cea72a5dd5, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.013106 6975 ts_tablet_manager.cc:1428] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:02:03.013062 6976 raft_consensus.cc:348] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.013198 6976 raft_consensus.cc:374] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.013242 6976 raft_consensus.cc:729] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d935e923dd408592a4313a8e478aba, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.013236 6699 heartbeater.cc:503] Master 127.1.217.126:41537 was elected leader, sending a full tablet report...
I20260926 07:02:03.013335 6976 consensus_queue.cc:260] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.013106 6977 consensus_queue.cc:260] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.013566 6961 heartbeater.cc:503] Master 127.1.217.126:41537 was elected leader, sending a full tablet report...
I20260926 07:02:03.013756 6977 ts_tablet_manager.cc:1428] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:02:03.013962 6830 heartbeater.cc:503] Master 127.1.217.126:41537 was elected leader, sending a full tablet report...
I20260926 07:02:03.014209 6976 ts_tablet_manager.cc:1428] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 07:02:03.032670 6982 raft_consensus.cc:482] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:03.032835 6982 raft_consensus.cc:504] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.033280 6982 leader_election.cc:288] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65a6ab073f304a28a8fa36f7fee2b8d3 (127.1.217.65:34563), e3677dc779234f49ab8789cea72a5dd5 (127.1.217.66:46239)
I20260926 07:02:03.034628 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.033442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34563 (local address 127.1.217.67:58335)
0926 07:02:03.033522 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.033534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.033579 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.033756 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.033758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.033760 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:03.034252 (+ 492us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:03.034253 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:03.034260 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:03.034273 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:03.034472 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:03.034473 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:03.034473 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 07:02:03.034545 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:02:03.034560 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 07:02:03.034629 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.033497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:58335 (local address 127.1.217.65:34563)
0926 07:02:03.033541 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.033543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.033576 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:02:03.033654 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.033655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.033670 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.033749 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.034368 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.034370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.034370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.034374 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.034397 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.034402 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.034468 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.034552 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.034567 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:02:03.034761 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.033516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:43283 (local address 127.1.217.66:46239)
0926 07:02:03.033570 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.033572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.033723 (+ 151us) server_negotiation.cc:407] Connection header received
0926 07:02:03.033787 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.033788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.033793 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.033849 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.034515 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.034516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.034517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.034521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.034534 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.034538 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.034590 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.034658 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.034670 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:02:03.034874 6984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.033459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46239 (local address 127.1.217.67:43283)
0926 07:02:03.033632 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.033641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.033722 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.033853 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.033854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.033872 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:03.034431 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:03.034431 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:03.034438 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:03.034451 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:03.034598 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:03.034599 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:03.034600 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 07:02:03.034650 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:02:03.034663 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260926 07:02:03.035184 6653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7418cc6a4942638e8d674e13f9ee43" candidate_uuid: "59d935e923dd408592a4313a8e478aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" is_pre_election: true
I20260926 07:02:03.035226 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7418cc6a4942638e8d674e13f9ee43" candidate_uuid: "59d935e923dd408592a4313a8e478aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3677dc779234f49ab8789cea72a5dd5" is_pre_election: true
I20260926 07:02:03.035341 6653 raft_consensus.cc:2454] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d935e923dd408592a4313a8e478aba in term 0.
I20260926 07:02:03.035364 6784 raft_consensus.cc:2454] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d935e923dd408592a4313a8e478aba in term 0.
I20260926 07:02:03.035605 6849 leader_election.cc:302] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d935e923dd408592a4313a8e478aba, 65a6ab073f304a28a8fa36f7fee2b8d3; no voters:
I20260926 07:02:03.035768 6982 raft_consensus.cc:2781] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:03.035815 6982 raft_consensus.cc:482] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:03.035830 6982 raft_consensus.cc:3037] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.036520 6982 raft_consensus.cc:504] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.036657 6982 leader_election.cc:288] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [CANDIDATE]: Term 1 election: Requested vote from peers 65a6ab073f304a28a8fa36f7fee2b8d3 (127.1.217.65:34563), e3677dc779234f49ab8789cea72a5dd5 (127.1.217.66:46239)
I20260926 07:02:03.036819 6653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7418cc6a4942638e8d674e13f9ee43" candidate_uuid: "59d935e923dd408592a4313a8e478aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3"
I20260926 07:02:03.036857 6784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7418cc6a4942638e8d674e13f9ee43" candidate_uuid: "59d935e923dd408592a4313a8e478aba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3677dc779234f49ab8789cea72a5dd5"
I20260926 07:02:03.036904 6653 raft_consensus.cc:3037] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.036926 6784 raft_consensus.cc:3037] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.037765 6784 raft_consensus.cc:2454] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d935e923dd408592a4313a8e478aba in term 1.
I20260926 07:02:03.037770 6653 raft_consensus.cc:2454] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d935e923dd408592a4313a8e478aba in term 1.
I20260926 07:02:03.037976 6849 leader_election.cc:302] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59d935e923dd408592a4313a8e478aba, 65a6ab073f304a28a8fa36f7fee2b8d3; no voters:
I20260926 07:02:03.038072 6982 raft_consensus.cc:2781] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:03.038250 6982 raft_consensus.cc:686] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 1 LEADER]: Becoming Leader. State: Replica: 59d935e923dd408592a4313a8e478aba, State: Running, Role: LEADER
I20260926 07:02:03.038477 6982 consensus_queue.cc:237] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } }
I20260926 07:02:03.039296 6509 catalog_manager.cc:5495] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba reported cstate change: term changed from 0 to 1, leader changed from <none> to 59d935e923dd408592a4313a8e478aba (127.1.217.67). New cstate: current_term: 1 leader_uuid: "59d935e923dd408592a4313a8e478aba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d935e923dd408592a4313a8e478aba" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46553 } health_report { overall_health: HEALTHY } } }
I20260926 07:02:03.044060 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.042958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.1.217.67:46553)
0926 07:02:03.043011 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.043013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.043045 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:02:03.043077 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.043078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.043091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.043141 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.043821 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.043823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.043823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.043828 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.043843 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.043847 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.043896 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.043980 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.043993 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:03.046116 6653 raft_consensus.cc:1261] T 1c7418cc6a4942638e8d674e13f9ee43 P 65a6ab073f304a28a8fa36f7fee2b8d3 [term 1 FOLLOWER]: Refusing update from remote peer 59d935e923dd408592a4313a8e478aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:03.046120 6784 raft_consensus.cc:1261] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 1 FOLLOWER]: Refusing update from remote peer 59d935e923dd408592a4313a8e478aba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:03.046454 6985 consensus_queue.cc:1035] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3677dc779234f49ab8789cea72a5dd5" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:03.046571 6982 consensus_queue.cc:1035] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [LEADER]: Connected to new peer: Peer: permanent_uuid: "65a6ab073f304a28a8fa36f7fee2b8d3" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:03.048801 6988 mvcc.cc:205] Tried to move back new op lower bound from 7333503479994421248 to 7333503479966420992. Current Snapshot: MvccSnapshot[applied={T|T < 7333503479994421248}]
I20260926 07:02:03.049126 6990 mvcc.cc:205] Tried to move back new op lower bound from 7333503479994421248 to 7333503479966420992. Current Snapshot: MvccSnapshot[applied={T|T < 7333503479994421248}]
I20260926 07:02:03.050863 6510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:03.050997 6510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:03.051708 6510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ee5c2597ce94075905264bddc6ffacb: Sending DeleteTablet for 3 replicas of tablet 1c7418cc6a4942638e8d674e13f9ee43
I20260926 07:02:03.052027 6633 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:03 UTC) from {username='slave'} at 127.0.0.1:39764
I20260926 07:02:03.052099 6764 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:03 UTC) from {username='slave'} at 127.0.0.1:60300
I20260926 07:02:03.052129 6895 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7418cc6a4942638e8d674e13f9ee43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:03 UTC) from {username='slave'} at 127.0.0.1:37354
I20260926 07:02:03.052310 6999 tablet_replica.cc:330] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: stopping tablet replica
I20260926 07:02:03.052325 7000 tablet_replica.cc:330] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: stopping tablet replica
I20260926 07:02:03.052404 7000 raft_consensus.cc:2229] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:03.052466 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6571
I20260926 07:02:03.052497 7000 raft_consensus.cc:2258] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:03.054101 7000 ts_tablet_manager.cc:1905] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:03.055197 6999 raft_consensus.cc:2229] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:03.055337 6999 raft_consensus.cc:2258] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:03.055742 6999 ts_tablet_manager.cc:1905] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:03.055759 7000 ts_tablet_manager.cc:1918] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:03.055822 7000 log.cc:1196] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/1c7418cc6a4942638e8d674e13f9ee43
I20260926 07:02:03.056104 7000 ts_tablet_manager.cc:1939] T 1c7418cc6a4942638e8d674e13f9ee43 P 59d935e923dd408592a4313a8e478aba: Deleting consensus metadata
I20260926 07:02:03.056622 6495 catalog_manager.cc:4841] TS 59d935e923dd408592a4313a8e478aba (127.1.217.67:46553): tablet 1c7418cc6a4942638e8d674e13f9ee43 (table test-table [id=7112050584884dee802625d826cde2ca]) successfully deleted
I20260926 07:02:03.057148 6999 ts_tablet_manager.cc:1918] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:03.057235 6999 log.cc:1196] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/1c7418cc6a4942638e8d674e13f9ee43
I20260926 07:02:03.057452 6999 ts_tablet_manager.cc:1939] T 1c7418cc6a4942638e8d674e13f9ee43 P e3677dc779234f49ab8789cea72a5dd5: Deleting consensus metadata
I20260926 07:02:03.057859 6495 catalog_manager.cc:4841] TS e3677dc779234f49ab8789cea72a5dd5 (127.1.217.66:46239): tablet 1c7418cc6a4942638e8d674e13f9ee43 (table test-table [id=7112050584884dee802625d826cde2ca]) successfully deleted
W20260926 07:02:03.063381 6496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:34563 (error 108)
I20260926 07:02:03.063723 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6702
I20260926 07:02:03.064446 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.063824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34563 (local address 127.0.0.1:39770)
0926 07:02:03.064344 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.064393 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496}
W20260926 07:02:03.064689 6496 catalog_manager.cc:4584] TS 65a6ab073f304a28a8fa36f7fee2b8d3 (127.1.217.65:34563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c7418cc6a4942638e8d674e13f9ee43: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34563: connect: Connection refused (error 111)
I20260926 07:02:03.068634 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6833
I20260926 07:02:03.072633 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 6480
2026-09-26T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (579 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 07:02:03.091845 1893 mini_oidc.cc:152] Starting JWKS server
I20260926 07:02:03.093047 1893 webserver.cc:460] Webserver started at https://127.0.0.1:39981/ using document root <none> and password file <none>
I20260926 07:02:03.093145 1893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:02:03.093278 1893 webserver.cc:460] Webserver started at http://127.0.0.1:37077/ using document root <none> and password file <none>
2026-09-26T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:03Z Disabled control of system clock
I20260926 07:02:03.101298 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42995
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:46473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42995
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39981/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/testchainca.cert with env {}
W20260926 07:02:03.180648 7007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:03.180868 7007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:03.180899 7007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:03.182053 7007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:03.182093 7007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:03.182106 7007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:03.182118 7007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:03.182158 7007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:02:03.182185 7007 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:02:03.182201 7007 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39981/jwks.json
I20260926 07:02:03.183672 7007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42995
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39981/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:03.183925 7007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:03.184139 7007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:03.189121 7013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.189129 7012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.189379 7007 server_base.cc:1034] running on GCE node
W20260926 07:02:03.189129 7015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.189697 7007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:03.190028 7007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:03.191198 7007 hybrid_clock.cc:648] HybridClock initialized: now 1790406123191149 us; error 60 us; skew 500 ppm
I20260926 07:02:03.192484 7007 webserver.cc:460] Webserver started at http://127.1.217.126:35891/ using document root <none> and password file <none>
I20260926 07:02:03.192761 7007 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:03.192876 7007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:03.193022 7007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:03.194187 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "24a832ef5da764335389fe1e1a4696d0"
server_key_iv: "4753027743d8f151caa57e77d461e1f0"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.194561 7007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "24a832ef5da764335389fe1e1a4696d0"
server_key_iv: "4753027743d8f151caa57e77d461e1f0"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.195837 7007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:02:03.196606 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:03.196837 7007 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:03.196950 7007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "24a832ef5da764335389fe1e1a4696d0"
server_key_iv: "4753027743d8f151caa57e77d461e1f0"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.197033 7007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:03.209837 7007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:03.210160 7007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:03.210309 7007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:03.214265 7007 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42995
I20260926 07:02:03.214309 7073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42995 every 8 connection(s)
I20260926 07:02:03.214679 7007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:03.215410 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7007
I20260926 07:02:03.215508 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:03.215757 1893 external_mini_cluster.cc:1428] Setting key 0e8218c5778d4e1979a3d434306cbcfa
I20260926 07:02:03.215864 7074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.219190 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a: Bootstrap starting.
I20260926 07:02:03.219864 7074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.220194 7074 log.cc:824] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.221061 7074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a: No bootstrap required, opened a new log
I20260926 07:02:03.221531 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.216318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.1.217.126:42995)
0926 07:02:03.216525 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.216532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.216557 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:02:03.217116 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.217124 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.217189 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.217308 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.217562 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.218444 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.219557 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.220053 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.220147 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.220824 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.220829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.220829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.221043 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.221085 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.221100 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.221172 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.221253 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.221326 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260926 07:02:03.222579 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:02:03.223234 7074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } }
I20260926 07:02:03.223408 7074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.223469 7074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b4b6235fe7a407c8e4f4bb80b3a0e7a, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.223590 7074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } }
I20260926 07:02:03.223685 7074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:03.223730 7074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:03.223781 7074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.224293 7074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } }
I20260926 07:02:03.224407 7074 leader_election.cc:302] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b4b6235fe7a407c8e4f4bb80b3a0e7a; no voters:
I20260926 07:02:03.224604 7074 leader_election.cc:288] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:03.224650 7079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:03.224864 7079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [term 1 LEADER]: Becoming Leader. State: Replica: 5b4b6235fe7a407c8e4f4bb80b3a0e7a, State: Running, Role: LEADER
I20260926 07:02:03.224920 7074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:03.224968 7079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } }
I20260926 07:02:03.225531 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } } }
I20260926 07:02:03.225526 7081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b4b6235fe7a407c8e4f4bb80b3a0e7a. Latest consensus state: current_term: 1 leader_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b4b6235fe7a407c8e4f4bb80b3a0e7a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42995 } } }
I20260926 07:02:03.225630 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.225634 7081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.225826 7088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:03.226357 7088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:03.227828 7088 catalog_manager.cc:1347] Generated new cluster ID: 4d0c861e2fe44286b4c9708a4df8b402
I20260926 07:02:03.227888 7088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:03.241405 7088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:03.242122 7088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:03.254242 7088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b4b6235fe7a407c8e4f4bb80b3a0e7a: Generated new TSK 0
I20260926 07:02:03.254494 7088 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:02:03.290566 7104 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 07:02:03.290685 7104 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:02:03.290910 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.271444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.1.217.126:42995)
0926 07:02:03.271495 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.271497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.271643 (+ 146us) server_negotiation.cc:407] Connection header received
0926 07:02:03.271743 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.271743 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.271754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.271813 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:02:03.271955 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.272431 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.273620 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.273798 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.273853 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.274080 (+ 227us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:02:03.290657 (+ 16577us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:02:03.290830 (+ 173us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 07:02:03.290987 7077 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 07:02:03.291301 7103 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:03.271345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42995 (local address 127.0.0.1:50964)
0926 07:02:03.271550 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.271562 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.271647 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.271820 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.271820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.271822 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:02:03.271888 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.271890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.272526 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.272527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.273520 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.273524 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.273880 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.273881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.273940 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.273960 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:02:03.290884 (+ 16924us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 07:02:03.291000 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.217.126:42995: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.1.217.126:42995: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39981/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 07:02:03.291715 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7007
2026-09-26T07:02:03Z chronyd exiting
I20260926 07:02:03.328068 1893 test_util.cc:175] -----------------------------------------------
I20260926 07:02:03.328145 1893 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790406101908495-1893-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (237 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 07:02:03.328986 1893 mini_oidc.cc:152] Starting JWKS server
I20260926 07:02:03.329627 1893 webserver.cc:460] Webserver started at https://127.0.0.1:46573/ using document root <none> and password file <none>
I20260926 07:02:03.329666 1893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:02:03.329769 1893 webserver.cc:460] Webserver started at http://127.0.0.1:35265/ using document root <none> and password file <none>
2026-09-26T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:03Z Disabled control of system clock
I20260926 07:02:03.337155 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:36097
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:43727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:36097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46573/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/testchainca.cert with env {}
W20260926 07:02:03.415462 7110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:03.415669 7110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:03.415690 7110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:03.416760 7110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:03.416817 7110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:03.416831 7110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:03.416842 7110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:03.416855 7110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:02:03.416867 7110 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:02:03.416878 7110 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46573/jwks.json
I20260926 07:02:03.418319 7110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:43727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:36097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:36097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46573/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:03.418509 7110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:03.418660 7110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:03.423418 7115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.423444 7118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.423460 7116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.423776 7110 server_base.cc:1034] running on GCE node
I20260926 07:02:03.423985 7110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:03.424216 7110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:03.425347 7110 hybrid_clock.cc:648] HybridClock initialized: now 1790406123425337 us; error 27 us; skew 500 ppm
I20260926 07:02:03.426553 7110 webserver.cc:460] Webserver started at http://127.1.217.126:38227/ using document root <none> and password file <none>
I20260926 07:02:03.426748 7110 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:03.426787 7110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:03.426872 7110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:03.427809 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "8848274f92b7c6c653c78565b05b5ab4"
server_key_iv: "8083117977dd3f8c84375698e5ccd5cf"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.428206 7110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "8848274f92b7c6c653c78565b05b5ab4"
server_key_iv: "8083117977dd3f8c84375698e5ccd5cf"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.429575 7110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:03.430235 7124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:03.430481 7110 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:03.430575 7110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "8848274f92b7c6c653c78565b05b5ab4"
server_key_iv: "8083117977dd3f8c84375698e5ccd5cf"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.430655 7110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:03.441303 7110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:03.441535 7110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:03.441629 7110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:03.445209 7110 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:36097
I20260926 07:02:03.445240 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:36097 every 8 connection(s)
I20260926 07:02:03.445554 7110 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:03.446422 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.448549 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16: Bootstrap starting.
I20260926 07:02:03.449142 7177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.449409 7177 log.cc:824] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.450050 7177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16: No bootstrap required, opened a new log
I20260926 07:02:03.450947 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7110
I20260926 07:02:03.451062 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:03.451254 1893 external_mini_cluster.cc:1428] Setting key a2620d65b89decec79edaf4f9a71709e
I20260926 07:02:03.452001 7177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } }
I20260926 07:02:03.452158 7177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.452191 7177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a6b6bc500ed4a5dbb9c18ae6349ef16, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.452322 7177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } }
I20260926 07:02:03.452401 7177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:03.452430 7177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:03.452489 7177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.453073 7177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } }
I20260926 07:02:03.453154 7177 leader_election.cc:302] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a6b6bc500ed4a5dbb9c18ae6349ef16; no voters:
I20260926 07:02:03.453322 7177 leader_election.cc:288] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:03.453344 7182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:03.453478 7182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [term 1 LEADER]: Becoming Leader. State: Replica: 7a6b6bc500ed4a5dbb9c18ae6349ef16, State: Running, Role: LEADER
I20260926 07:02:03.453562 7177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:03.453581 7182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } }
I20260926 07:02:03.453991 7182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a6b6bc500ed4a5dbb9c18ae6349ef16. Latest consensus state: current_term: 1 leader_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } } }
I20260926 07:02:03.454001 7183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a6b6bc500ed4a5dbb9c18ae6349ef16" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 36097 } } }
I20260926 07:02:03.454180 7183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.454092 7182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.454938 7191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:03.455623 7191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:03.457086 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.451715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.1.217.126:36097)
0926 07:02:03.451932 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.451941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.451973 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:02:03.452077 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.452083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.452147 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.452280 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.452576 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.454105 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.455182 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.455602 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.455737 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.456369 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.456372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.456372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.456566 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.456612 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.456622 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.456709 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.456798 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.456884 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 07:02:03.458038 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:02:03.461189 7191 catalog_manager.cc:1347] Generated new cluster ID: 0ba8b23336f348a5923f9577c2d4e958
I20260926 07:02:03.461248 7191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:03.470063 7191 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:03.470551 7191 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:03.473737 7191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a6b6bc500ed4a5dbb9c18ae6349ef16: Generated new TSK 0
I20260926 07:02:03.473909 7191 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 07:02:03.492863 7207 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 07:02:03.492949 7207 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 07:02:03.493239 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.478820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.1.217.126:36097)
0926 07:02:03.478879 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.478881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.478960 (+ 79us) server_negotiation.cc:407] Connection header received
0926 07:02:03.479000 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.479001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.479013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.479103 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:02:03.479206 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.479975 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.480797 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.481117 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.481191 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.481391 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:02:03.493020 (+ 11629us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:02:03.493169 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:02:03.493297 7181 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 07:02:03.493319 7206 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:03.478742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:36097 (local address 127.0.0.1:45276)
0926 07:02:03.478893 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.478902 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.478956 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.479100 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.479101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.479102 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:02:03.479150 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.479151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.480089 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.480090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.480687 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.480689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.481218 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.481219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.481277 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.481282 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:02:03.493134 (+ 11852us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 07:02:03.493228 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.217.126:36097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46573/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260926 07:02:03.493597 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7110
2026-09-26T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (202 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 07:02:03.531024 1893 mini_oidc.cc:152] Starting JWKS server
I20260926 07:02:03.531726 1893 webserver.cc:460] Webserver started at https://127.0.0.1:40771/ using document root <none> and password file <none>
I20260926 07:02:03.531796 1893 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 07:02:03.531986 1893 webserver.cc:460] Webserver started at http://127.0.0.1:35851/ using document root <none> and password file <none>
2026-09-26T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:03Z Disabled control of system clock
I20260926 07:02:03.539433 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:40427
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:33481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:40427
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40771/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:02:03.616787 7213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:03.616984 7213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:03.617004 7213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:03.618162 7213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:03.618202 7213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:03.618214 7213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:03.618227 7213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:03.618239 7213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:02:03.618295 7213 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:02:03.618310 7213 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40771/jwks.json
I20260926 07:02:03.619609 7213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:40427
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:40427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40771/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:03.619855 7213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:03.620129 7213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:03.624852 7219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.624853 7221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.624897 7218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.625075 7213 server_base.cc:1034] running on GCE node
I20260926 07:02:03.625340 7213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:03.625604 7213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:03.626765 7213 hybrid_clock.cc:648] HybridClock initialized: now 1790406123626756 us; error 39 us; skew 500 ppm
I20260926 07:02:03.628037 7213 webserver.cc:460] Webserver started at http://127.1.217.126:42621/ using document root <none> and password file <none>
I20260926 07:02:03.628338 7213 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:03.628414 7213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:03.628589 7213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:03.629837 7213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "22de7a7867614557b7cf0ae1a301cfc7"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "27b06049f77fcc1cac17971b995f3ae7"
server_key_iv: "f79218950b868bfe1b54a1ad7c3462ee"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.630226 7213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "22de7a7867614557b7cf0ae1a301cfc7"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "27b06049f77fcc1cac17971b995f3ae7"
server_key_iv: "f79218950b868bfe1b54a1ad7c3462ee"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.631641 7213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:03.632467 7227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:03.632644 7213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:03.632761 7213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "22de7a7867614557b7cf0ae1a301cfc7"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "27b06049f77fcc1cac17971b995f3ae7"
server_key_iv: "f79218950b868bfe1b54a1ad7c3462ee"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.632912 7213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:03.656575 7213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:03.657013 7213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:03.657178 7213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:03.661072 7213 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:40427
I20260926 07:02:03.661110 7279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:40427 every 8 connection(s)
I20260926 07:02:03.661455 7213 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:03.662120 7280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.663359 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7213
I20260926 07:02:03.663460 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:03.663666 1893 external_mini_cluster.cc:1428] Setting key 0d9a4a63dd55e636863dbd31b37510cd
I20260926 07:02:03.665580 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7: Bootstrap starting.
I20260926 07:02:03.666244 7280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.666540 7280 log.cc:824] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.667335 7280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7: No bootstrap required, opened a new log
I20260926 07:02:03.669721 7280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } }
I20260926 07:02:03.669865 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.664253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.1.217.126:40427)
0926 07:02:03.664463 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.664471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.664498 (+ 27us) server_negotiation.cc:407] Connection header received
0926 07:02:03.665350 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.665358 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.665426 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.665556 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.665823 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.666873 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.667969 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.668382 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.668471 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.669233 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.669236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.669237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.669403 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.669435 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.669445 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.669496 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.669591 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.669684 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260926 07:02:03.669871 7280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.670006 7280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22de7a7867614557b7cf0ae1a301cfc7, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.670136 7280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } }
I20260926 07:02:03.670219 7280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:03.670254 7280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:03.670325 7280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.670837 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:02:03.671000 7280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } }
I20260926 07:02:03.671097 7280 leader_election.cc:302] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22de7a7867614557b7cf0ae1a301cfc7; no voters:
I20260926 07:02:03.671237 7280 leader_election.cc:288] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:03.671267 7285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:03.671428 7285 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [term 1 LEADER]: Becoming Leader. State: Replica: 22de7a7867614557b7cf0ae1a301cfc7, State: Running, Role: LEADER
I20260926 07:02:03.671481 7280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:03.671509 7285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } }
I20260926 07:02:03.671835 7286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22de7a7867614557b7cf0ae1a301cfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } } }
I20260926 07:02:03.671928 7286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.672092 7287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22de7a7867614557b7cf0ae1a301cfc7. Latest consensus state: current_term: 1 leader_uuid: "22de7a7867614557b7cf0ae1a301cfc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22de7a7867614557b7cf0ae1a301cfc7" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 40427 } } }
I20260926 07:02:03.672123 7291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:03.672155 7287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.672610 7291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:03.674409 7291 catalog_manager.cc:1347] Generated new cluster ID: e2307addf4d6434aac62256a1b82f01b
I20260926 07:02:03.674475 7291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:03.692014 7291 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:03.692584 7291 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:03.701241 7291 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22de7a7867614557b7cf0ae1a301cfc7: Generated new TSK 0
I20260926 07:02:03.701438 7291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:03.711694 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.707954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.1.217.126:40427)
0926 07:02:03.708010 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.708012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.708169 (+ 157us) server_negotiation.cc:407] Connection header received
0926 07:02:03.708241 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.708241 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.708251 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.708309 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:02:03.708451 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.709006 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.710144 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.710341 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.710411 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.710629 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:02:03.711465 (+ 836us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:02:03.711614 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 07:02:03.711755 7283 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60666: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 07:02:03.711776 7309 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:03.707866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:40427 (local address 127.0.0.1:60666)
0926 07:02:03.708088 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.708099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.708173 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.708327 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.708328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.708330 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:02:03.708389 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.708391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.709079 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.709080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.710044 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.710047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.710428 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.710430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.710492 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.710498 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:02:03.711577 (+ 1079us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 07:02:03.711669 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.217.126:40427: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40771/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260926 07:02:03.712069 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7213
E20260926 07:02:03.726084 7310 webserver.cc:533] Webserver: sslize failed:
E20260926 07:02:03.726176 7310 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T07:02:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:03Z Disabled control of system clock
I20260926 07:02:03.756295 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42955
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:34023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42955
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/testchainca.cert with env {}
W20260926 07:02:03.835248 7314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:03.835455 7314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:03.835477 7314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:03.836551 7314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:03.836585 7314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:03.836598 7314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:03.836611 7314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:03.836622 7314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 07:02:03.836635 7314 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 07:02:03.836647 7314 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 07:02:03.838059 7314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:34023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:03.838248 7314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:03.838395 7314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:03.843299 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:03.843304 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.843380 7314 server_base.cc:1034] running on GCE node
W20260926 07:02:03.843317 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:03.843818 7314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:03.844116 7314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:03.845264 7314 hybrid_clock.cc:648] HybridClock initialized: now 1790406123845243 us; error 34 us; skew 500 ppm
I20260926 07:02:03.846495 7314 webserver.cc:460] Webserver started at http://127.1.217.126:42505/ using document root <none> and password file <none>
I20260926 07:02:03.846740 7314 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:03.846791 7314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:03.846940 7314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:03.848037 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "66dbd6fc5e044d4dbb918044670118e5"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "78652b127398707fc44310bc74bfa0cd"
server_key_iv: "db7f40fb530f9e7527bca352488e8648"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.848418 7314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "66dbd6fc5e044d4dbb918044670118e5"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "78652b127398707fc44310bc74bfa0cd"
server_key_iv: "db7f40fb530f9e7527bca352488e8648"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.849931 7314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:03.850736 7328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:03.850963 7314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:03.851059 7314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "66dbd6fc5e044d4dbb918044670118e5"
format_stamp: "Formatted at 2026-09-26 07:02:03 on dist-test-slave-t1mc"
server_key: "78652b127398707fc44310bc74bfa0cd"
server_key_iv: "db7f40fb530f9e7527bca352488e8648"
server_key_version: "encryptionkey@0"
I20260926 07:02:03.851135 7314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:03.863934 7314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:03.864233 7314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:03.864390 7314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:03.868391 7314 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42955
I20260926 07:02:03.868440 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42955 every 8 connection(s)
I20260926 07:02:03.868793 7314 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:03.869692 7381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:03.870463 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7314
I20260926 07:02:03.870548 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:03.870721 1893 external_mini_cluster.cc:1428] Setting key 524f013859b25a55ee693a965e958ae7
I20260926 07:02:03.872687 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5: Bootstrap starting.
I20260926 07:02:03.873276 7381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:03.873598 7381 log.cc:824] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:03.874338 7381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5: No bootstrap required, opened a new log
I20260926 07:02:03.876463 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.871258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.1.217.126:42955)
0926 07:02:03.871455 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.871472 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.871496 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:02:03.871931 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.871938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.872007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.872122 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:03.872373 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.873469 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.874528 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.874973 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.875068 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.875740 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:03.875743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:03.875743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:03.875946 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:03.875979 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:03.875990 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:03.876042 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:03.876156 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:02:03.876215 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260926 07:02:03.876566 7381 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } }
I20260926 07:02:03.876730 7381 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:03.876756 7381 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66dbd6fc5e044d4dbb918044670118e5, State: Initialized, Role: FOLLOWER
I20260926 07:02:03.877067 7381 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } }
I20260926 07:02:03.877157 7381 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:03.877199 7381 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:03.877252 7381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:03.877549 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:02:03.877899 7381 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } }
I20260926 07:02:03.878016 7381 leader_election.cc:302] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66dbd6fc5e044d4dbb918044670118e5; no voters:
I20260926 07:02:03.878203 7381 leader_election.cc:288] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:03.878232 7386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:03.878454 7381 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:03.878763 7386 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [term 1 LEADER]: Becoming Leader. State: Replica: 66dbd6fc5e044d4dbb918044670118e5, State: Running, Role: LEADER
I20260926 07:02:03.878867 7386 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } }
I20260926 07:02:03.879269 7387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66dbd6fc5e044d4dbb918044670118e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } } }
I20260926 07:02:03.879281 7388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66dbd6fc5e044d4dbb918044670118e5. Latest consensus state: current_term: 1 leader_uuid: "66dbd6fc5e044d4dbb918044670118e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66dbd6fc5e044d4dbb918044670118e5" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42955 } } }
I20260926 07:02:03.879493 7388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.879448 7387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:03.879863 7399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:03.880488 7399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:03.882035 7399 catalog_manager.cc:1347] Generated new cluster ID: ed068f3c1c394a7f825e316c63d4f185
I20260926 07:02:03.882087 7399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:03.892437 7399 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:03.893036 7399 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:03.901124 7399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66dbd6fc5e044d4dbb918044670118e5: Generated new TSK 0
I20260926 07:02:03.901351 7399 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:03.960186 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:03.916302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.1.217.126:42955)
0926 07:02:03.916371 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:03.916373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:03.916512 (+ 139us) server_negotiation.cc:407] Connection header received
0926 07:02:03.916583 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:03.916584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:03.916595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:03.916657 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0926 07:02:03.916866 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.917369 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.918526 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.918721 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.918815 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.918968 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 07:02:03.959977 (+ 41009us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:02:03.960125 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34600: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 07:02:03.960273 7384 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34600: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 07:02:03.960457 7410 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:03.916255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42955 (local address 127.0.0.1:34600)
0926 07:02:03.916434 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:03.916445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:03.916512 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:03.916673 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:03.916674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:03.916676 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 07:02:03.916741 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.916743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.917442 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.917443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.918416 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:03.918420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:03.918806 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:03.918807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:03.918867 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:03.918872 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 07:02:03.960124 (+ 41252us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 07:02:03.960265 (+ 141us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.1.217.126:42955: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
I20260926 07:02:03.960834 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7314
2026-09-26T07:02:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (228 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:04 dist-test-slave-t1mc krb5kdc[7422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:04 dist-test-slave-t1mc krb5kdc[7422](info): set up 2 sockets
Sep 26 07:02:04 dist-test-slave-t1mc krb5kdc[7422](info): commencing operation
krb5kdc: starting...
W20260926 07:02:05.168987 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:02:05.287705 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45149
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:45149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:05.366585 7435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:05.366786 7435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:05.366807 7435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:05.367859 7435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:05.367893 7435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:05.367906 7435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:05.367919 7435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:05.367930 7435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:05.369357 7435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:45149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:05.369544 7435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:05.369714 7435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:05.374468 7443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.374408 7441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.374418 7440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.374630 7435 server_base.cc:1034] running on GCE node
I20260926 07:02:05.374967 7435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:05.375299 7435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:05.376438 7435 hybrid_clock.cc:648] HybridClock initialized: now 1790406125376422 us; error 28 us; skew 500 ppm
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.378930 7435 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:02:05.379369 7435 webserver.cc:460] Webserver started at http://127.1.217.126:33189/ using document root <none> and password file <none>
I20260926 07:02:05.379613 7435 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:05.379659 7435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:05.379817 7435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:05.380981 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "90e0da821ca3414fb110ebf229a0eb02"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "ce10a0f0941ca3f396577c44da52c9b9"
server_key_iv: "548dbeb5cacf3a7a8c1628270fc10394"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.381332 7435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "90e0da821ca3414fb110ebf229a0eb02"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "ce10a0f0941ca3f396577c44da52c9b9"
server_key_iv: "548dbeb5cacf3a7a8c1628270fc10394"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.382635 7435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:05.383322 7450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.383486 7435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:05.383590 7435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "90e0da821ca3414fb110ebf229a0eb02"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "ce10a0f0941ca3f396577c44da52c9b9"
server_key_iv: "548dbeb5cacf3a7a8c1628270fc10394"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.383668 7435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:05.397111 7435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:05.398895 7435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:05.399062 7435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:05.402995 7435 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:45149
I20260926 07:02:05.403030 7502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:45149 every 8 connection(s)
I20260926 07:02:05.403375 7435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:05.404243 7503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:05.406467 7503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02: Bootstrap starting.
I20260926 07:02:05.406972 7503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:05.407202 7503 log.cc:824] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:05.407872 7503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02: No bootstrap required, opened a new log
I20260926 07:02:05.409487 7503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } }
I20260926 07:02:05.409619 7503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:05.409641 7503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e0da821ca3414fb110ebf229a0eb02, State: Initialized, Role: FOLLOWER
I20260926 07:02:05.409814 7503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } }
I20260926 07:02:05.409894 7503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:05.409916 7503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:05.409945 7503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:05.410398 7503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } }
I20260926 07:02:05.410480 7503 leader_election.cc:302] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90e0da821ca3414fb110ebf229a0eb02; no voters:
I20260926 07:02:05.410597 7503 leader_election.cc:288] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:05.410669 7506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:05.410840 7503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:05.411060 7506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [term 1 LEADER]: Becoming Leader. State: Replica: 90e0da821ca3414fb110ebf229a0eb02, State: Running, Role: LEADER
I20260926 07:02:05.411162 7506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } }
I20260926 07:02:05.411561 7507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90e0da821ca3414fb110ebf229a0eb02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } } }
I20260926 07:02:05.411661 7507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:05.411809 7508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90e0da821ca3414fb110ebf229a0eb02. Latest consensus state: current_term: 1 leader_uuid: "90e0da821ca3414fb110ebf229a0eb02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90e0da821ca3414fb110ebf229a0eb02" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45149 } } }
I20260926 07:02:05.411883 7508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:05.411861 7515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:05.412091 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7435
I20260926 07:02:05.412221 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:05.412469 1893 external_mini_cluster.cc:1428] Setting key e43a8adabe3689d9bc7d566ef078e393
I20260926 07:02:05.412773 7515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:05.414399 7515 catalog_manager.cc:1347] Generated new cluster ID: bb504bdf526d4f9799ec26bb67a6e3e1
I20260926 07:02:05.414451 7515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:05.422459 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.413039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.1.217.126:45149)
0926 07:02:05.413218 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.413227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.413309 (+ 82us) server_negotiation.cc:407] Connection header received
0926 07:02:05.413471 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.413478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.413536 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.413692 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.414397 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.415389 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.416145 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.416567 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.416738 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.419122 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.419126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.419126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.419145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.420884 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.421282 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.421285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.421286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.421385 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.421602 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.421603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.421604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.421839 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.421995 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.422108 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.422254 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260926 07:02:05.424674 7515 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:05.425215 7515 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:05.426915 7515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90e0da821ca3414fb110ebf229a0eb02: Generated new TSK 0
I20260926 07:02:05.427075 7515 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:02:05.455618 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:05.534945 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:05.535168 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:05.535208 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:05.536413 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:05.536480 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:05.536527 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:05.538024 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:05.538270 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:05.538496 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:05.543380 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.543376 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.543556 7529 server_base.cc:1034] running on GCE node
W20260926 07:02:05.543397 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.543813 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:05.544054 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:05.545208 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790406125545201 us; error 47 us; skew 500 ppm
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.547602 7529 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:02:05.548002 7529 webserver.cc:460] Webserver started at http://127.1.217.65:35107/ using document root <none> and password file <none>
I20260926 07:02:05.548214 7529 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:05.548286 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:05.548432 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:05.549505 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "f60d56e1191d401386c7ef875e9ef91b"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "295a115a1e9c068384b4720ae5c01dd7"
server_key_iv: "78a855a2d7245c95986877a8a48a365f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.549855 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "f60d56e1191d401386c7ef875e9ef91b"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "295a115a1e9c068384b4720ae5c01dd7"
server_key_iv: "78a855a2d7245c95986877a8a48a365f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.551110 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:02:05.551787 7544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.551980 7529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:05.552078 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "f60d56e1191d401386c7ef875e9ef91b"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "295a115a1e9c068384b4720ae5c01dd7"
server_key_iv: "78a855a2d7245c95986877a8a48a365f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.552165 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:05.564083 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:05.566408 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:05.566566 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:05.566794 7529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:05.567099 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:05.567153 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.567195 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:05.567222 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.572929 7529 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:38063
I20260926 07:02:05.572983 7657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:38063 every 8 connection(s)
I20260926 07:02:05.573328 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:05.580467 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7529
I20260926 07:02:05.580582 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:05.580824 1893 external_mini_cluster.cc:1428] Setting key 03703b7034b62ca9ae9e5820cfea37fd
I20260926 07:02:05.582690 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.574043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:53193 (local address 127.1.217.126:45149)
0926 07:02:05.574104 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.574106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.574288 (+ 182us) server_negotiation.cc:407] Connection header received
0926 07:02:05.575199 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.575200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.575210 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.575261 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.575844 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.576339 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.577394 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.577590 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.577644 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.579644 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.579646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.579647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.579657 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.581248 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.581738 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.581739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.581739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.581810 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.582059 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.582060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.582061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.582173 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.582247 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.582518 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.582600 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 07:02:05.583060 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.574016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45149 (local address 127.1.217.65:53193)
0926 07:02:05.574170 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:05.574213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:05.575104 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:05.575323 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:05.575326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:05.575570 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:05.575773 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.575776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.576419 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.576420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.577323 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.577325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.577662 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.577662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.577769 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.578229 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:05.578237 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:05.579545 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:05.581336 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.581343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.581345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.581650 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.581875 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.581876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.581876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.581991 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.582238 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:05.582244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:05.582389 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 07:02:05.582498 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:02:05.582653 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 07:02:05.583578 7658 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45149
I20260926 07:02:05.583683 7658 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:05.583824 7658 heartbeater.cc:511] Master 127.1.217.126:45149 requested a full tablet report, sending...
I20260926 07:02:05.584271 7467 ts_manager.cc:194] Registered new tserver with Master: f60d56e1191d401386c7ef875e9ef91b (127.1.217.65:38063)
I20260926 07:02:05.585139 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:53193
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:02:05.607575 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:05.689041 7663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:05.689240 7663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:05.689263 7663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:05.690420 7663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:05.690462 7663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:05.690488 7663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:05.692060 7663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:05.692291 7663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:05.692536 7663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:05.697572 7669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.697578 7668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.697608 7671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.697692 7663 server_base.cc:1034] running on GCE node
I20260926 07:02:05.697889 7663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:05.698135 7663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:05.699295 7663 hybrid_clock.cc:648] HybridClock initialized: now 1790406125699278 us; error 31 us; skew 500 ppm
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.701931 7663 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:02:05.702354 7663 webserver.cc:460] Webserver started at http://127.1.217.66:45487/ using document root <none> and password file <none>
I20260926 07:02:05.702567 7663 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:05.702636 7663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:05.702786 7663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:05.703775 7663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "32f460922ad3408abbffa4272427932d"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "37bafde7b5706ee239d03b7fe482a6fe"
server_key_iv: "b4ac7c84efb49c031070511a1f50ea1e"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.704120 7663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "32f460922ad3408abbffa4272427932d"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "37bafde7b5706ee239d03b7fe482a6fe"
server_key_iv: "b4ac7c84efb49c031070511a1f50ea1e"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.705542 7663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:05.706327 7678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.706529 7663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:05.706666 7663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "32f460922ad3408abbffa4272427932d"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "37bafde7b5706ee239d03b7fe482a6fe"
server_key_iv: "b4ac7c84efb49c031070511a1f50ea1e"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.706744 7663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:05.715083 7663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:05.717342 7663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:05.717499 7663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:05.717723 7663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:05.718005 7663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:05.718058 7663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.718098 7663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:05.718127 7663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.723846 7663 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:33529
I20260926 07:02:05.723917 7791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:33529 every 8 connection(s)
I20260926 07:02:05.724227 7663 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:05.732270 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7663
I20260926 07:02:05.732357 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:05.732573 1893 external_mini_cluster.cc:1428] Setting key 1d90d7cd9f5a44c813fa1155cea88cd4
I20260926 07:02:05.733717 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.725052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:34589 (local address 127.1.217.126:45149)
0926 07:02:05.725099 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.725101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.725325 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:02:05.726277 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.726279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.726290 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.726345 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.726930 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.727434 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.728434 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.728636 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.728733 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.730907 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.730909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.730909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.730917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.732105 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.732604 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.732605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.732605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.732687 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.732955 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.732957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.732957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.733110 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.733178 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.733517 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.733606 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:02:05.733793 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.725027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45149 (local address 127.1.217.66:34589)
0926 07:02:05.725206 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:05.725240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:05.726152 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:05.726410 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:05.726414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:05.726656 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:05.726834 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.726838 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.727528 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.727529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.728338 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.728340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.728742 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.728743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.728894 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.729366 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:05.729374 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:05.730809 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:05.732211 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.732217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.732220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.732517 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.732741 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.732742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.732742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.732885 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.733197 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:05.733203 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:05.733364 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 07:02:05.733473 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:02:05.733599 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260926 07:02:05.734534 7792 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45149
I20260926 07:02:05.734651 7792 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:05.734822 7792 heartbeater.cc:511] Master 127.1.217.126:45149 requested a full tablet report, sending...
I20260926 07:02:05.735224 7467 ts_manager.cc:194] Registered new tserver with Master: 32f460922ad3408abbffa4272427932d (127.1.217.66:33529)
I20260926 07:02:05.735591 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:34589
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:02:05.757925 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:05.838958 7797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:05.839154 7797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:05.839176 7797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:05.840359 7797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:05.840404 7797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:05.840430 7797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:05.841996 7797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:05.842231 7797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:05.842468 7797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:05.847348 7803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.847393 7797 server_base.cc:1034] running on GCE node
W20260926 07:02:05.847363 7802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:05.847438 7805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:05.847755 7797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:05.847986 7797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:05.849149 7797 hybrid_clock.cc:648] HybridClock initialized: now 1790406125849142 us; error 35 us; skew 500 ppm
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.851603 7797 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:02:05.852094 7797 webserver.cc:460] Webserver started at http://127.1.217.67:45003/ using document root <none> and password file <none>
I20260926 07:02:05.852311 7797 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:05.852380 7797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:05.852527 7797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:05.853566 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "eb31793a4e8e4634b6f87696f427e4ca"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "814a03eb095d476b6e5f6b07d6cd5a25"
server_key_iv: "5df096f70f15e02ad785ae17c86b701f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.853932 7797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "eb31793a4e8e4634b6f87696f427e4ca"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "814a03eb095d476b6e5f6b07d6cd5a25"
server_key_iv: "5df096f70f15e02ad785ae17c86b701f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.855297 7797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:02:05.856014 7812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.856184 7797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:05.856280 7797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "eb31793a4e8e4634b6f87696f427e4ca"
format_stamp: "Formatted at 2026-09-26 07:02:05 on dist-test-slave-t1mc"
server_key: "814a03eb095d476b6e5f6b07d6cd5a25"
server_key_iv: "5df096f70f15e02ad785ae17c86b701f"
server_key_version: "encryptionkey@0"
I20260926 07:02:05.856366 7797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:05.877260 7797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:05.880213 7797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:05.880375 7797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:05.880630 7797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:05.881023 7797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:05.881078 7797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.881115 7797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:05.881142 7797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:05.887964 7797 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:44615
I20260926 07:02:05.888036 7925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:44615 every 8 connection(s)
I20260926 07:02:05.888278 7797 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:05.892621 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7797
I20260926 07:02:05.892760 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:05.893302 1893 external_mini_cluster.cc:1428] Setting key ab6029c123776d414475412dfce7700f
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:05.897171 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.889045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:51859 (local address 127.1.217.126:45149)
0926 07:02:05.889101 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.889103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.889284 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:02:05.890183 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.890184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.890193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.890255 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.890800 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.891268 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.892229 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.892409 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.892463 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.894888 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.894889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.894889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.894895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.896043 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.896403 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.896404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.896404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.896450 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.896618 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.896619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.896619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.896723 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.896812 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.897054 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.897109 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:02:05.897356 7928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.889021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45149 (local address 127.1.217.67:51859)
0926 07:02:05.889164 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:05.889205 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:05.890099 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:05.890311 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:05.890314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:05.890549 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:05.890737 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.890741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.891339 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.891340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.892152 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:05.892153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.892492 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.892493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:05.892641 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.893223 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:05.893235 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:05.894814 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:05.896115 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.896121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.896123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.896342 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.896490 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:05.896490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:05.896490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:05.896572 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:05.896822 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:05.896828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:05.896945 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 07:02:05.897030 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 07:02:05.897139 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260926 07:02:05.898751 7926 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45149
I20260926 07:02:05.898844 7926 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:05.898985 7926 heartbeater.cc:511] Master 127.1.217.126:45149 requested a full tablet report, sending...
I20260926 07:02:05.899333 7467 ts_manager.cc:194] Registered new tserver with Master: eb31793a4e8e4634b6f87696f427e4ca (127.1.217.67:44615)
I20260926 07:02:05.899703 7467 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:51859
I20260926 07:02:05.905207 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/test-admin.keytab.
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.922170 1893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 07:02:05.923794 7933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:05.923066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.1.217.65:38063)
0926 07:02:05.923235 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.923240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.923250 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:05.923275 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.923277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.923290 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.923357 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.923698 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46874: BlockingRecv error: recv got EOF from 127.0.0.1:46874 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260926 07:02:05.923875 1893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.924945 1893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.925792 1893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:02:05.933997 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.926273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46876 (local address 127.1.217.65:38063)
0926 07:02:05.926331 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.926334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.926402 (+ 68us) server_negotiation.cc:407] Connection header received
0926 07:02:05.926483 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.926484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.926495 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.926568 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.927032 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.927855 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.928607 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.929090 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.929160 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.931412 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.931421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.931421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.931428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.932815 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.933157 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.933160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.933161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.933224 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.933424 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.933425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.933425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.933565 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.933765 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.933811 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.933887 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 07:02:05.935734 7933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:05.935252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46878 (local address 127.1.217.65:38063)
0926 07:02:05.935308 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.935310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.935318 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:05.935367 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.935368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.935375 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.935425 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.935647 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46878: BlockingRecv error: recv got EOF from 127.0.0.1:46878 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:02:05.935798 1893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.936681 1893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.937566 1893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:02:05.944211 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.938044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46884 (local address 127.1.217.65:38063)
0926 07:02:05.938094 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.938096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.938140 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:02:05.938209 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.938210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.938219 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.938272 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.938621 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.939073 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.939804 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.940020 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.940086 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.941986 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.941987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.941988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.941994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.943131 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.943496 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.943497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.943497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.943551 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.943744 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.943745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.943746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.943844 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.943942 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.944045 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.944116 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 07:02:05.945664 7933 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:05.945149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.1.217.65:38063)
0926 07:02:05.945202 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.945204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.945229 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:02:05.945298 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.945299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.945308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.945361 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.945618 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46890: BlockingRecv error: recv got EOF from 127.0.0.1:46890 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:02:05.945747 1893 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790406101908495-1893-0/krb5kdc/krb5cc' of type FILE
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.946648 1893 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:05.947458 1893 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 07:02:05 dist-test-slave-t1mc krb5kdc[7422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:02:05.954205 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:05.947909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.1.217.65:38063)
0926 07:02:05.947961 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:05.947964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:05.948012 (+ 48us) server_negotiation.cc:407] Connection header received
0926 07:02:05.948084 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:05.948085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:05.948093 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:05.948147 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:05.948504 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.949065 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.949770 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:05.949958 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:05.950040 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:05.951884 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:05.951885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:05.951885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:05.951892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:05.953109 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.953457 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.953458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.953458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.953514 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:05.953719 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:05.953720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:05.953720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:05.953813 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:05.953921 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:05.954026 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 07:02:05.954096 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:02:05.954806 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7529
I20260926 07:02:05.959172 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7663
I20260926 07:02:05.963193 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7797
I20260926 07:02:05.967176 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7435
2026-09-26T07:02:05Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1833 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:06 dist-test-slave-t1mc krb5kdc[7946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:06 dist-test-slave-t1mc krb5kdc[7946](info): set up 2 sockets
Sep 26 07:02:06 dist-test-slave-t1mc krb5kdc[7946](info): commencing operation
krb5kdc: starting...
W20260926 07:02:07.001021 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab.
Entry for principal oryx/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab.
I20260926 07:02:07.119560 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:38595
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab
--principal=oryx/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:38595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:07.198743 7959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:07.198966 7959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:07.199002 7959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:07.200111 7959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:07.200176 7959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:07.200201 7959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:07.200234 7959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:07.200263 7959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:07.201746 7959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:38595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab
--principal=oryx/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:38595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:07.202008 7959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:07.202239 7959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.207486 7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.207475 7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.207475 7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.207643 7959 server_base.cc:1034] running on GCE node
I20260926 07:02:07.207907 7959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.208233 7959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.209388 7959 hybrid_clock.cc:648] HybridClock initialized: now 1790406127209378 us; error 28 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.211748 7959 init.cc:377] Logged in from keytab as oryx/127.1.217.126@KRBTEST.COM (short username oryx)
I20260926 07:02:07.212208 7959 webserver.cc:460] Webserver started at http://127.1.217.126:43183/ using document root <none> and password file <none>
I20260926 07:02:07.212426 7959 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.212502 7959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.212651 7959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:07.213728 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "b9d6220d324442efb2c4200fa924ac73"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "f2e7737d64a8a28821028e1bf66b119b"
server_key_iv: "b809e96e70cae95607b99f34c11bbb35"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.214088 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "b9d6220d324442efb2c4200fa924ac73"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "f2e7737d64a8a28821028e1bf66b119b"
server_key_iv: "b809e96e70cae95607b99f34c11bbb35"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.215406 7959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:07.216149 7974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.216349 7959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:07.216447 7959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "b9d6220d324442efb2c4200fa924ac73"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "f2e7737d64a8a28821028e1bf66b119b"
server_key_iv: "b809e96e70cae95607b99f34c11bbb35"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.216528 7959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.228332 7959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.230283 7959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:07.230441 7959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.234407 7959 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:38595
I20260926 07:02:07.234457 8026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:38595 every 8 connection(s)
I20260926 07:02:07.234822 7959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:07.235767 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:07.238198 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Bootstrap starting.
I20260926 07:02:07.238755 8027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:07.239037 8027 log.cc:824] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:07.239689 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: No bootstrap required, opened a new log
I20260926 07:02:07.241257 8027 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } }
I20260926 07:02:07.241384 8027 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:07.241431 8027 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d6220d324442efb2c4200fa924ac73, State: Initialized, Role: FOLLOWER
I20260926 07:02:07.241580 8027 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } }
I20260926 07:02:07.241637 8027 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:07.241657 8027 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:07.241726 8027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:07.242491 8027 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } }
I20260926 07:02:07.242597 8027 leader_election.cc:302] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9d6220d324442efb2c4200fa924ac73; no voters:
I20260926 07:02:07.242772 8027 leader_election.cc:288] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:07.242820 8030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:07.242985 8030 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 LEADER]: Becoming Leader. State: Replica: b9d6220d324442efb2c4200fa924ac73, State: Running, Role: LEADER
I20260926 07:02:07.243052 8027 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:07.243069 8030 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } }
I20260926 07:02:07.243505 8031 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } }
I20260926 07:02:07.243613 8031 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:07.243775 8032 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d6220d324442efb2c4200fa924ac73. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } }
I20260926 07:02:07.243819 8039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:07.243844 8032 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:07.244311 8039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:07.244338 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 7959
I20260926 07:02:07.244446 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:07.244688 1893 external_mini_cluster.cc:1428] Setting key d8cd59574e8288a20b28a431dc413bb1
I20260926 07:02:07.246039 8039 catalog_manager.cc:1347] Generated new cluster ID: 7c9dece23a06422896704a4cc0192c29
I20260926 07:02:07.246094 8039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:07.254218 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.245190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.1.217.126:38595)
0926 07:02:07.245348 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.245357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.245388 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:02:07.245533 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.245540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.245611 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.245767 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.246478 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.247371 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.248135 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.248554 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.248683 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.250987 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.250990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.250991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.251002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.252690 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.253067 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.253070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.253070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.253182 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.253392 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.253392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.253393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.253633 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.253812 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.253888 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.254025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260926 07:02:07.274422 8039 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:07.275014 8039 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:07.280704 8039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Generated new TSK 0
I20260926 07:02:07.280928 8039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
I20260926 07:02:07.311564 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:07.390326 8053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:07.390558 8053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:07.390595 8053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:07.391722 8053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:07.391800 8053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:07.391845 8053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:07.393301 8053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:07.393566 8053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:07.393791 8053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.398700 8058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.398691 8059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.398805 8053 server_base.cc:1034] running on GCE node
W20260926 07:02:07.398691 8061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.399143 8053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.399384 8053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.400559 8053 hybrid_clock.cc:648] HybridClock initialized: now 1790406127400554 us; error 37 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.403163 8053 init.cc:377] Logged in from keytab as oryx/127.1.217.65@KRBTEST.COM (short username oryx)
I20260926 07:02:07.403604 8053 webserver.cc:460] Webserver started at http://127.1.217.65:38367/ using document root <none> and password file <none>
I20260926 07:02:07.403795 8053 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.403837 8053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.403970 8053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:07.405064 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "62c25104d4b141ccbe833b90edb35618"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "5280e130655f02151026225f007e1da5"
server_key_iv: "ed0aec46888bbca853e5f666ac1889d5"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.405409 8053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "62c25104d4b141ccbe833b90edb35618"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "5280e130655f02151026225f007e1da5"
server_key_iv: "ed0aec46888bbca853e5f666ac1889d5"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.406760 8053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:07.407503 8068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.407684 8053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:07.407791 8053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "62c25104d4b141ccbe833b90edb35618"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "5280e130655f02151026225f007e1da5"
server_key_iv: "ed0aec46888bbca853e5f666ac1889d5"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.407866 8053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.418684 8053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.421074 8053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:07.421233 8053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.421458 8053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:07.421762 8053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:07.421816 8053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.421857 8053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:07.421885 8053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.427759 8053 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43787
I20260926 07:02:07.427820 8181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43787 every 8 connection(s)
I20260926 07:02:07.428153 8053 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:07.436455 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8053
I20260926 07:02:07.436554 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:07.436833 1893 external_mini_cluster.cc:1428] Setting key 78aacb1a4f75283f3a0c08752a54378f
I20260926 07:02:07.437487 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.428946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:35239 (local address 127.1.217.126:38595)
0926 07:02:07.429016 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.429019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.429211 (+ 192us) server_negotiation.cc:407] Connection header received
0926 07:02:07.429671 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.429671 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.429684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.429737 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.430623 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.431121 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.432134 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.432336 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.432391 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.434376 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.434377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.434377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.434383 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.435882 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.436307 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.436308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.436308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.436377 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.436705 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.436707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.436707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.436883 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.436958 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.437305 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.437397 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:07.437685 8184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.428924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:35239)
0926 07:02:07.429090 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.429125 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.429590 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.429788 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.429791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.430356 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.430543 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.430547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.431217 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.431218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.432049 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.432051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.432412 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.432413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.432509 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.432999 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.433007 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.434294 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.435966 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.435972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.435974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.436188 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.436458 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.436458 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.436458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.436639 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.436961 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.436966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.437126 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.437275 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.437426 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260926 07:02:07.438100 8182 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:07.438189 8182 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:07.438380 8182 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:07.438922 7991 ts_manager.cc:194] Registered new tserver with Master: 62c25104d4b141ccbe833b90edb35618 (127.1.217.65:43787)
I20260926 07:02:07.439626 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:35239
WARNING: no policy specified for oryx/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
I20260926 07:02:07.461756 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:07.540691 8187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:07.540928 8187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:07.540967 8187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:07.542107 8187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:07.542171 8187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:07.542213 8187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:07.543699 8187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:07.543948 8187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:07.544165 8187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.549098 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.549089 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.549077 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.549227 8187 server_base.cc:1034] running on GCE node
I20260926 07:02:07.549463 8187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.549746 8187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.550911 8187 hybrid_clock.cc:648] HybridClock initialized: now 1790406127550897 us; error 47 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.553354 8187 init.cc:377] Logged in from keytab as oryx/127.1.217.66@KRBTEST.COM (short username oryx)
I20260926 07:02:07.553815 8187 webserver.cc:460] Webserver started at http://127.1.217.66:44537/ using document root <none> and password file <none>
I20260926 07:02:07.554033 8187 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.554101 8187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.554248 8187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:07.555243 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "c8acfe0b9053c6cf83eace12976dd43c"
server_key_iv: "4e15c7b34305fb79f374ef1ad8f9ab6b"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.555601 8187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "c8acfe0b9053c6cf83eace12976dd43c"
server_key_iv: "4e15c7b34305fb79f374ef1ad8f9ab6b"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.556965 8187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:02:07.557652 8202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.557812 8187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.557910 8187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "c8acfe0b9053c6cf83eace12976dd43c"
server_key_iv: "4e15c7b34305fb79f374ef1ad8f9ab6b"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.557997 8187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.583309 8187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.585325 8187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:07.585486 8187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.585736 8187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:07.586090 8187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:07.586143 8187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.586186 8187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:07.586213 8187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.591883 8187 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:43059
I20260926 07:02:07.591929 8315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:43059 every 8 connection(s)
I20260926 07:02:07.592289 8187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:07.596544 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8187
I20260926 07:02:07.596668 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:07.596894 1893 external_mini_cluster.cc:1428] Setting key e286d421ba79ece5a9c0e438bd47fe16
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:07.602118 8318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.593049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:41459)
0926 07:02:07.593203 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.593245 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.593726 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.593958 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.593962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.594529 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.594724 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.594728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.595395 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.595396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.596190 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.596192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.596663 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.596664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.596851 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.597444 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.597457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.598996 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.600821 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.600827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.600829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.601049 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.601275 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.601276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.601276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.601370 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.601652 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.601656 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.601764 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.601844 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.601956 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 07:02:07.602129 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.593089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:41459 (local address 127.1.217.126:38595)
0926 07:02:07.593137 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.593139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.593323 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:02:07.593823 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.593824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.593834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.593900 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.594810 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.595317 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.596274 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.596567 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.596647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.599107 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.599108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.599108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.599114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.600702 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.601132 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.601134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.601135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.601202 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.601440 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.601441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.601442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.601585 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.601657 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.601892 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.601991 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:07.602643 8316 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:07.602754 8316 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:07.602970 8316 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:07.603376 7991 ts_manager.cc:194] Registered new tserver with Master: 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:07.603750 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:41459
WARNING: no policy specified for oryx/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
I20260926 07:02:07.624018 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:07.703598 8321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:07.703799 8321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:07.703822 8321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:07.704984 8321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:07.705027 8321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:07.705053 8321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:07.706451 8321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:38595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:07.706686 8321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:07.706921 8321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.711939 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.711947 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.711947 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.712342 8321 server_base.cc:1034] running on GCE node
I20260926 07:02:07.712589 8321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.712845 8321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.714008 8321 hybrid_clock.cc:648] HybridClock initialized: now 1790406127713976 us; error 45 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.716440 8321 init.cc:377] Logged in from keytab as oryx/127.1.217.67@KRBTEST.COM (short username oryx)
I20260926 07:02:07.716982 8321 webserver.cc:460] Webserver started at http://127.1.217.67:37381/ using document root <none> and password file <none>
I20260926 07:02:07.717234 8321 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.717304 8321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.717447 8321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:07.718461 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "ddb801c7a3c0cfd2ed6f60e01530b2a1"
server_key_iv: "c0a69e006251c813ed64057398c69d62"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.718878 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "ddb801c7a3c0cfd2ed6f60e01530b2a1"
server_key_iv: "c0a69e006251c813ed64057398c69d62"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.720212 8321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:02:07.720947 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.721118 8321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:07.721215 8321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "ddb801c7a3c0cfd2ed6f60e01530b2a1"
server_key_iv: "c0a69e006251c813ed64057398c69d62"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.721292 8321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.730145 8321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.732039 8321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:07.732189 8321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.732400 8321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:07.732708 8321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:07.732764 8321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.732823 8321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:07.732853 8321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.738612 8321 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:37337
I20260926 07:02:07.738659 8449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:37337 every 8 connection(s)
I20260926 07:02:07.738981 8321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:07.747718 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.739708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:35895 (local address 127.1.217.126:38595)
0926 07:02:07.739783 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.739785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.739938 (+ 153us) server_negotiation.cc:407] Connection header received
0926 07:02:07.740409 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.740411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.740420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.740470 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.741479 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.741957 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.742999 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.743192 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.743246 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.745457 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.745471 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.745472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.745478 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.746649 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.746983 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.746983 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.746984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.747038 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.747234 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.747235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.747235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.747327 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.747376 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.747599 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.747655 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:02:07.747967 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.739674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:35895)
0926 07:02:07.739829 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.739868 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.740329 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.740538 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.740542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.741220 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.741403 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.741407 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.742032 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.742033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.742913 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.742915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.743260 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.743261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.743353 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.743892 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.743902 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.745381 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.746718 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.746723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.746725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.746922 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.747093 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.747094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.747094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.747183 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.747379 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.747383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.747482 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.747572 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.747680 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 07:02:07.748449 8450 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:07.748493 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8321
I20260926 07:02:07.748553 8450 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:07.748592 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:07.748742 8450 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:07.748816 1893 external_mini_cluster.cc:1428] Setting key f7922bed89eae5f8c7454aca3f1a988b
I20260926 07:02:07.749179 7990 ts_manager.cc:194] Registered new tserver with Master: daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337)
I20260926 07:02:07.749548 7990 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:35895
I20260926 07:02:07.750252 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.217.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.125.keytab.
Entry for principal oryx/127.1.217.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.125.keytab.
I20260926 07:02:07.775385 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34917
--webserver_interface=127.1.217.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.125.keytab
--principal=oryx/127.1.217.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:38595,127.1.217.125:34917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:07.855350 8455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:07.855559 8455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:07.855580 8455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:07.856632 8455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:07.856666 8455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:07.856680 8455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:07.856693 8455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:07.856703 8455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:07.858160 8455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:38595,127.1.217.125:34917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.125.keytab
--principal=oryx/127.1.217.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.217.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
W20260926 07:02:07.858251 8455 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.1.217.126:38595,127.1.217.125:34917'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 07:02:07.858352 8455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:07.858505 8455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.863097 8463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.863111 8460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.863346 8455 server_base.cc:1034] running on GCE node
W20260926 07:02:07.863112 8461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.863616 8455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.863850 8455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.864980 8455 hybrid_clock.cc:648] HybridClock initialized: now 1790406127864985 us; error 45 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.867373 8455 init.cc:377] Logged in from keytab as oryx/127.1.217.125@KRBTEST.COM (short username oryx)
I20260926 07:02:07.867838 8455 webserver.cc:460] Webserver started at http://127.1.217.125:45645/ using document root <none> and password file <none>
I20260926 07:02:07.868057 8455 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.868124 8455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.868273 8455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:07.869385 8455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/instance:
uuid: "317927315e3044b9bb73241a4e5b0d4a"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "57e7373a1366c306ce38282b227f66d4"
server_key_iv: "85699563e6dfb4c546822fc23222cc3e"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.869745 8455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal/instance:
uuid: "317927315e3044b9bb73241a4e5b0d4a"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "57e7373a1366c306ce38282b227f66d4"
server_key_iv: "85699563e6dfb4c546822fc23222cc3e"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.871018 8455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:07.871786 8470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.871994 8455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:07.872092 8455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "317927315e3044b9bb73241a4e5b0d4a"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "57e7373a1366c306ce38282b227f66d4"
server_key_iv: "85699563e6dfb4c546822fc23222cc3e"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.872174 8455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.880330 8455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.882525 8455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:07.882679 8455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.886569 8455 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.125:34917
I20260926 07:02:07.886597 8522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.125:34917 every 8 connection(s)
I20260926 07:02:07.886920 8455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
I20260926 07:02:07.887804 8523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:07.889267 8523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:02:07.889638 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8455
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.125@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
I20260926 07:02:07.898524 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.889959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.1.217.126:38595)
0926 07:02:07.890035 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.890037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.890397 (+ 360us) server_negotiation.cc:407] Connection header received
0926 07:02:07.890880 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.890881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.890894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.890952 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.891472 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.891966 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.892910 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.893099 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.893186 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.895459 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.895461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.895461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.895468 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.897096 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.897558 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.897559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.897560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.897620 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.897862 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.897863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.897864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.898016 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.898087 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.898344 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.898419 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:02:07.898830 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.889957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45564)
0926 07:02:07.890296 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.890323 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.890787 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.890971 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.890974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.891240 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.891377 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.891381 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.892078 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.892079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.892809 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.892811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.893209 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.893210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.893331 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.893816 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.893825 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.895373 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.897193 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.897199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.897201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.897469 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.897686 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.897687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.897687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.897793 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.898090 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.898096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.898228 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.898323 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.898467 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":105,"threads_started":1}
I20260926 07:02:07.898833 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.890132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44564 (local address 127.1.217.125:34917)
0926 07:02:07.890298 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.890309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.890367 (+ 58us) server_negotiation.cc:407] Connection header received
0926 07:02:07.890802 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.890811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.890831 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.890926 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.891552 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.892345 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.893171 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.893456 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.893533 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.895955 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.895962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.895962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.895969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.897446 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.897811 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.897814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.897815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.897869 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.898072 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.898073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.898073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.898232 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.898380 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.898471 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.898549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260926 07:02:07.899410 8523 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.125@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
I20260926 07:02:07.904896 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.899660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44566)
0926 07:02:07.899691 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.899701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.899744 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.899842 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.899843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.899941 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.899967 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.899968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.900536 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.900536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.901062 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.901064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.901380 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.901381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.901419 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.901807 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.901811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.902846 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.904058 (+ 1212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.904059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.904059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.904204 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.904373 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.904373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.904373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.904432 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.904619 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.904620 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.904650 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.904699 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.904805 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 07:02:07.904943 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.899685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.1.217.125:34917)
0926 07:02:07.899730 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.899732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.899740 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:07.899796 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.899797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.899805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.899841 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.900025 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.900465 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.901131 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.901319 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.901370 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.902913 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.902914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.902914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.902920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.903993 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.904265 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.904266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.904267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.904317 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.904470 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.904470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.904471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.904569 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.904616 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.904712 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.904853 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":29}
I20260926 07:02:07.906119 8523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Bootstrap starting.
I20260926 07:02:07.906855 8523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:07.907076 8523 log.cc:824] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:07.907760 8523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: No bootstrap required, opened a new log
I20260926 07:02:07.909228 8523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } }
I20260926 07:02:07.909345 8523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:07.909368 8523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317927315e3044b9bb73241a4e5b0d4a, State: Initialized, Role: FOLLOWER
I20260926 07:02:07.909520 8523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } }
I20260926 07:02:07.909757 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } } }
I20260926 07:02:07.909811 8523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:07.909830 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:02:07.911437 8540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:02:07.911496 8540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:02:07.912981 8455 master_runner.cc:428] Detected that this master 317927315e3044b9bb73241a4e5b0d4a is joining an existing cluster
I20260926 07:02:07.913029 8455 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.217.125:34917
I20260926 07:02:07.919760 8529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:07.919860 8529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } }
I20260926 07:02:07.920048 8529 leader_election.cc:288] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595)
I20260926 07:02:07.920256 8001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "317927315e3044b9bb73241a4e5b0d4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d6220d324442efb2c4200fa924ac73" is_pre_election: true
I20260926 07:02:07.920710 8471 leader_election.cc:302] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 317927315e3044b9bb73241a4e5b0d4a; no voters: b9d6220d324442efb2c4200fa924ac73
I20260926 07:02:07.920902 8529 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 07:02:07.921502 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.914201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44568 (local address 127.1.217.125:34917)
0926 07:02:07.914233 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.914235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.914539 (+ 304us) server_negotiation.cc:407] Connection header received
0926 07:02:07.914581 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.914582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.914592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.914651 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.914992 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.915611 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.916293 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.916532 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.916588 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.917608 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.917609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.917609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.917615 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.919274 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920404 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.920405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.920406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.920466 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920661 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.920662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.920662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.920833 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.920878 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.921253 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.921382 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":18}
I20260926 07:02:07.921540 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.914004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.1.217.126:38595)
0926 07:02:07.914043 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.914045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.914182 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:02:07.914286 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.914287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.914298 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.914343 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.914721 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.915407 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.916471 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.916791 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.917141 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.918317 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.918318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.918319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.918582 (+ 263us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.920264 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920671 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.920672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.920672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.920737 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.921021 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.921022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.921022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.921150 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.921191 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.921367 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.921442 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:07.921838 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.913939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45568)
0926 07:02:07.914106 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.914114 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.914207 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.914356 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.914357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.914546 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.914633 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.914634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.915490 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.915491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.916390 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.916392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.917219 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.917221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.917288 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.917800 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.917803 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.918226 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.920370 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.920371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.920590 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.920867 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.920868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.920942 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.921235 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.921235 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.921270 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.921331 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.921444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":21,"thread_start_us":84,"threads_started":1}
I20260926 07:02:07.922175 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.914157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44568)
0926 07:02:07.914460 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.914471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.914529 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.914672 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.914672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.914845 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.914904 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.914906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.915682 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.915684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.916200 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.916202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.916639 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.916640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.916690 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.917129 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.917134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.917522 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.919344 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.919345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.919345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.919575 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.920544 (+ 969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.920545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.920545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.920598 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.921118 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.921119 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.921159 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.921228 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.921457 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":55,"threads_started":1}
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.126@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
I20260926 07:02:07.930876 8527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.924068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.1.217.125:34917)
0926 07:02:07.924100 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.924102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.924263 (+ 161us) server_negotiation.cc:407] Connection header received
0926 07:02:07.924349 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.924350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.924359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.924425 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.924849 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.925289 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.926055 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.926249 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.926330 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.928340 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.928342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.928342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.928348 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.929718 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.930100 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.930101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.930101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.930158 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.930353 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.930354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.930355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.930459 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.930546 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.930718 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.930788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:07.930912 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.924018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44580)
0926 07:02:07.924161 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.924189 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.924287 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.924431 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.924435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.924695 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.924735 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.924737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.925370 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.925371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.925951 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.925953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.926333 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.926334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.926426 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.926846 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.926850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.928256 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.929817 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.929824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.929826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.930026 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.930223 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.930225 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.930225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.930282 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.930542 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.930549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.930596 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.930694 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.930801 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260926 07:02:07.931470 7991 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.217.125:34917: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a"
member_type: NON_VOTER
last_known_addr {
host: "127.1.217.125"
port: 34917
}
attrs {
promote: true
}
}
dest_uuid: "b9d6220d324442efb2c4200fa924ac73"
cas_config_opid_index: -1
I20260926 07:02:07.931780 7991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } }
I20260926 07:02:07.932231 8549 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 317927315e3044b9bb73241a4e5b0d4a (127.1.217.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } }
I20260926 07:02:07.932504 8497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:07.933350 8549 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.217.125:34917
I20260926 07:02:07.933373 8497 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 07:02:07.933571 8455 master.cc:537] Master@127.1.217.125:34917 shutting down...
I20260926 07:02:07.933609 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } }
I20260926 07:02:07.933686 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:07.933662 8549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 07:02:07.934365 7975 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 07:02:07.934759 7975 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 -> Peer 317927315e3044b9bb73241a4e5b0d4a (127.1.217.125:34917): Couldn't send request to peer 317927315e3044b9bb73241a4e5b0d4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 07:02:07.936580 8455 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:07.936661 8455 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:07.936723 8455 tablet_replica.cc:330] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: stopping tablet replica
I20260926 07:02:07.949512 8455 master.cc:552] Master@127.1.217.125:34917 shutdown complete.
I20260926 07:02:07.949554 8455 master_runner.cc:305] Clearing existing system tablet
I20260926 07:02:07.949690 8455 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.949738 8455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.950456 8455 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.002s sys 0.000s
I20260926 07:02:07.950948 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.951064 8455 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.951120 8455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.951143 8455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "317927315e3044b9bb73241a4e5b0d4a"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "57e7373a1366c306ce38282b227f66d4"
server_key_iv: "85699563e6dfb4c546822fc23222cc3e"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.951726 8455 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:07.953090 8455 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:02:07.953145 8455 log.cc:1196] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 07:02:07.953336 8455 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Deleting consensus metadata
I20260926 07:02:07.953460 8455 master_runner.cc:315] Copying system tablet from 127.1.217.126:38595
I20260926 07:02:07.954020 8455 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: tablet copy: Beginning tablet copy session from remote peer at address 127.1.217.126:38595
I20260926 07:02:07.960011 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.954364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.1.217.126:38595)
0926 07:02:07.954409 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:07.954411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:07.954511 (+ 100us) server_negotiation.cc:407] Connection header received
0926 07:02:07.954591 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:07.954592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:07.954603 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:07.954660 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:07.954974 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.955467 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.956195 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.956404 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.956495 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.957472 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:07.957474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:07.957474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:07.957480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:07.958886 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.959246 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.959247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.959247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.959304 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:07.959498 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:07.959499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:07.959500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:07.959607 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:07.959683 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:07.959839 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 07:02:07.959914 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:07.960064 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:07.954281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45578)
0926 07:02:07.954429 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:07.954439 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:07.954527 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:07.954677 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:07.954677 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:07.954841 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:07.954894 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.954895 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.955557 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.955558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.956109 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:07.956110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:07.956490 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:07.956490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:07.956555 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:07.957072 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:07.957077 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:07.957365 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:07.958994 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.958995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.958995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.959165 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.959368 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:07.959369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:07.959369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:07.959425 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:07.959674 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:07.959675 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:07.959718 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:02:07.959805 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:02:07.959918 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260926 07:02:07.960551 8011 tablet_copy_service.cc:140] P b9d6220d324442efb2c4200fa924ac73: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 317927315e3044b9bb73241a4e5b0d4a ({username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:45578)
I20260926 07:02:07.960619 8011 tablet_copy_service.cc:161] P b9d6220d324442efb2c4200fa924ac73: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 317927315e3044b9bb73241a4e5b0d4a at {username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:45578: session id = 317927315e3044b9bb73241a4e5b0d4a-00000000000000000000000000000000
I20260926 07:02:07.961347 8011 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:02:07.961949 8455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:02:07.962805 8455 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: tablet copy: Starting download of 0 data blocks...
I20260926 07:02:07.962949 8455 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: tablet copy: Starting download of 1 WAL segments...
I20260926 07:02:07.963742 8455 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:02:07.964601 8011 tablet_copy_service.cc:342] P b9d6220d324442efb2c4200fa924ac73: Request end of tablet copy session 317927315e3044b9bb73241a4e5b0d4a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:45578
I20260926 07:02:07.964658 8011 tablet_copy_service.cc:433] P b9d6220d324442efb2c4200fa924ac73: ending tablet copy session 317927315e3044b9bb73241a4e5b0d4a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 317927315e3044b9bb73241a4e5b0d4a
I20260926 07:02:07.966944 8455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:07.968039 8562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.968026 8561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:07.968272 8564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:07.968302 8455 server_base.cc:1034] running on GCE node
I20260926 07:02:07.968446 8455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:07.968590 8455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:07.969712 8455 hybrid_clock.cc:648] HybridClock initialized: now 1790406127969704 us; error 23 us; skew 500 ppm
Sep 26 07:02:07 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:07.971129 8455 init.cc:377] Logged in from keytab as oryx/127.1.217.125@KRBTEST.COM (short username oryx)
I20260926 07:02:07.971400 8455 webserver.cc:460] Webserver started at http://127.1.217.125:36027/ using document root <none> and password file <none>
I20260926 07:02:07.971535 8455 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:07.971575 8455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:07.972154 8455 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:07.972652 8571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.972815 8455 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:07.972857 8455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "317927315e3044b9bb73241a4e5b0d4a"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "57e7373a1366c306ce38282b227f66d4"
server_key_iv: "85699563e6dfb4c546822fc23222cc3e"
server_key_version: "encryptionkey@0"
I20260926 07:02:07.972930 8455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:07.978413 8455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:07.979986 8455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:07.982995 8455 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.125:34917
I20260926 07:02:07.983239 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.125:34917 every 8 connection(s)
I20260926 07:02:07.983392 8455 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
I20260926 07:02:07.984052 8624 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:02:07.984458 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Bootstrap starting.
I20260926 07:02:07.986722 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:02:07.986943 8624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Bootstrap complete.
I20260926 07:02:07.987133 8624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } }
I20260926 07:02:07.987215 8624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 317927315e3044b9bb73241a4e5b0d4a, State: Initialized, Role: LEARNER
I20260926 07:02:07.987265 8624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } }
I20260926 07:02:07.987454 8624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:07.987449 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } }
I20260926 07:02:07.987566 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: LEARNER
I20260926 07:02:07.988863 8638 catalog_manager.cc:1259] Loaded cluster ID: 7c9dece23a06422896704a4cc0192c29
I20260926 07:02:07.988916 8638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: loading cluster ID for follower catalog manager: success
I20260926 07:02:07.989997 8638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: acquiring CA information for follower catalog manager: success
I20260926 07:02:07.990306 8638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:02:08.007002 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.000499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.1.217.125:34917)
0926 07:02:08.000621 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.000624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.000632 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:08.000661 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.000661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.000670 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.000746 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.001318 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.001941 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.002655 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.002853 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.002937 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.004136 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.004138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.004138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.004145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.005812 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.006143 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.006144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.006145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.006212 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.006397 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.006398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.006398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.006532 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.006638 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.006721 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.006800 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.1.217.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.124.keytab.
Entry for principal oryx/127.1.217.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.124.keytab.
I20260926 07:02:08.032266 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:38647
--webserver_interface=127.1.217.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.124.keytab
--principal=oryx/127.1.217.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:08.114166 8642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:08.114388 8642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:08.114425 8642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:08.115566 8642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:08.115630 8642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:08.115655 8642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:08.115687 8642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:08.115711 8642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:08.117236 8642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.124.keytab
--principal=oryx/127.1.217.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.217.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:08.117504 8642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:08.117720 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:08.122749 8648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.122807 8642 server_base.cc:1034] running on GCE node
W20260926 07:02:08.122778 8650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:08.122749 8647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.123214 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:08.123519 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:08.124660 8642 hybrid_clock.cc:648] HybridClock initialized: now 1790406128124648 us; error 30 us; skew 500 ppm
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:08.127246 8642 init.cc:377] Logged in from keytab as oryx/127.1.217.124@KRBTEST.COM (short username oryx)
I20260926 07:02:08.127681 8642 webserver.cc:460] Webserver started at http://127.1.217.124:43083/ using document root <none> and password file <none>
I20260926 07:02:08.127897 8642 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.127940 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.128108 8642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:08.129235 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/instance:
uuid: "21339434331f4ebcac499ec595ee3935"
format_stamp: "Formatted at 2026-09-26 07:02:08 on dist-test-slave-t1mc"
server_key: "32971bf1490ae83f28fa19b5f5bba538"
server_key_iv: "17856875dd7c6055aa35338a4d437db3"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.129593 8642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal/instance:
uuid: "21339434331f4ebcac499ec595ee3935"
format_stamp: "Formatted at 2026-09-26 07:02:08 on dist-test-slave-t1mc"
server_key: "32971bf1490ae83f28fa19b5f5bba538"
server_key_iv: "17856875dd7c6055aa35338a4d437db3"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.130867 8642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:08.131565 8657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.131758 8642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:08.131844 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "21339434331f4ebcac499ec595ee3935"
format_stamp: "Formatted at 2026-09-26 07:02:08 on dist-test-slave-t1mc"
server_key: "32971bf1490ae83f28fa19b5f5bba538"
server_key_iv: "17856875dd7c6055aa35338a4d437db3"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.131914 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.160357 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:08.162767 8642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:08.162928 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:08.166744 8642 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.124:38647
I20260926 07:02:08.166785 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.124:38647 every 8 connection(s)
I20260926 07:02:08.167135 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
I20260926 07:02:08.167414 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8642
I20260926 07:02:08.168352 8710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:08.170344 8710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.124@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:08.178535 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.170962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45582)
0926 07:02:08.171099 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.171136 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.171197 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.171357 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.171360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.171616 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.171676 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.171678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.172365 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.172366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.173199 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.173202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.173571 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.173573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.173667 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.174176 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.174184 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.175585 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.177373 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.177379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.177650 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.177835 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.177836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.177885 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.178152 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.178158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.178209 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.178297 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.178390 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260926 07:02:08.178541 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.168052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.1.217.124:38647)
0926 07:02:08.168257 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.168267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.168300 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:02:08.168939 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.168948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.169026 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.169171 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.169716 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.171052 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.171843 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.172242 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.172355 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.175060 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.175066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.175066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.175072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.177013 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177332 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.177335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.177336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.177431 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177640 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.177641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.177641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.177941 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.178080 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.178150 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.178262 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
I20260926 07:02:08.178591 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.171140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.1.217.126:38595)
0926 07:02:08.171194 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.171196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.171206 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:08.171256 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.171257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.171267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.171338 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.171765 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.172243 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.173286 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.173471 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.173526 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.175671 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.175672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.175672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.175680 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.177276 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177719 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.177720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.177720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.177773 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.177945 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.177946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.177946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.178072 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.178148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.178420 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.178494 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:02:08.179518 8710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.124@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
I20260926 07:02:08.185000 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.179743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44596)
0926 07:02:08.179787 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.179797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.179830 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.179967 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.179967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.180069 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.180108 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.180109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.180686 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.180687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.181217 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.181218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.181532 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.181533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.181579 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.181951 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.181956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.182972 (+ 1016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.184221 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.184221 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.184222 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.184364 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.184527 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.184527 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.184528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.184573 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.184743 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.184743 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.184814 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.184875 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.184937 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 07:02:08.185014 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.179815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.1.217.125:34917)
0926 07:02:08.179853 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.179855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.179862 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:08.179887 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.179888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.179902 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.179955 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.180169 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.180625 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.181282 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.181473 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.181524 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.183039 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.183040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.183040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.183045 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.184156 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.184428 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.184428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.184428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.184473 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.184612 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.184612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.184613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.184696 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.184740 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.184895 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.184954 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:08.185392 8710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.124@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
I20260926 07:02:08.191030 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.185631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54852)
0926 07:02:08.185678 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.185688 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.185728 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.185846 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.185846 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.185971 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.185993 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.185994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.186607 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.186608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.187059 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.187061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.187505 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.187506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.187553 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.187923 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.187927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.189060 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.190280 (+ 1220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.190281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.190281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.190429 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.190589 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.190589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.190589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.190636 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.190801 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.190801 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.190831 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.190879 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.190944 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:02:08.191111 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.185674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54852 (local address 127.1.217.124:38647)
0926 07:02:08.185708 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.185710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.185717 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:08.185783 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.185783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.185791 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.185842 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.186066 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.186528 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.187151 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.187428 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.187506 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.189132 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.189133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.189133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.189138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.190215 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.190490 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.190491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.190491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.190534 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.190673 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.190673 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.190674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.190753 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.190796 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.190893 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.190985 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":21}
I20260926 07:02:08.192382 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Bootstrap starting.
I20260926 07:02:08.193281 8710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:08.193559 8710 log.cc:824] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:08.194258 8710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: No bootstrap required, opened a new log
I20260926 07:02:08.196584 8710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } }
I20260926 07:02:08.196807 8710 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:08.196856 8710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21339434331f4ebcac499ec595ee3935, State: Initialized, Role: FOLLOWER
I20260926 07:02:08.196983 8710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } }
I20260926 07:02:08.197335 8710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:08.197330 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } } }
I20260926 07:02:08.197453 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:02:08.199262 8726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:02:08.199357 8726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:02:08.201661 8642 master_runner.cc:428] Detected that this master 21339434331f4ebcac499ec595ee3935 is joining an existing cluster
I20260926 07:02:08.201709 8642 master_runner.cc:432] Initiating AddMaster RPC to add 127.1.217.124:38647
I20260926 07:02:08.209619 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.202612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.1.217.126:38595)
0926 07:02:08.202655 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.202657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.202855 (+ 198us) server_negotiation.cc:407] Connection header received
0926 07:02:08.202879 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.202880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.202891 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.202938 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.203799 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.204600 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.205698 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.205982 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206020 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.207157 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.207158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.207158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.207165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.208480 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.208856 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.208857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.208857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.208910 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.209132 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.209133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.209134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.209247 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.209323 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.209451 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.209526 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:08.209633 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.202562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45588)
0926 07:02:08.202721 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.202738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.202808 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.202987 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.202988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.203624 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.203698 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.203701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.204686 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.204686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.205603 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.205605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.206105 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.206178 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.206711 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.206716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.207042 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.208575 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.208576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.208577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.208748 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.209001 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.209002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.209002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.209060 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.209313 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.209314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.209355 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.209425 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.209528 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406127, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.126@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
I20260926 07:02:08.229089 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.203003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.1.217.124:38647)
0926 07:02:08.203033 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.203035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.203685 (+ 650us) server_negotiation.cc:407] Connection header received
0926 07:02:08.203779 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.203780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.203787 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.203813 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.204189 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.204924 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.205947 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.206262 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206303 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.226086 (+ 19783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.226087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.226087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.226094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.227844 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228274 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228324 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228524 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228651 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.228691 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.228909 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.228986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 07:02:08.229131 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.202862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.1.217.125:34917)
0926 07:02:08.202911 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.202913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.203477 (+ 564us) server_negotiation.cc:407] Connection header received
0926 07:02:08.203574 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.203575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.203585 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.203616 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.204004 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.204725 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.205819 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.206113 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206179 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.226106 (+ 19927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.226107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.226108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.226115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.227903 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228342 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228344 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228396 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228590 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228712 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.228752 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.228973 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.229043 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:02:08.229203 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.211938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54876)
0926 07:02:08.211990 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.212001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.212043 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.217024 (+ 4981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.217026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.217166 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.217202 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.217204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.217840 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.217841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.218334 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.218335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.218709 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.218710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.218778 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.219219 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.219224 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.220628 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.228098 (+ 7470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.228100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228314 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228528 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.228529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228582 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228888 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.228889 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.228935 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.228981 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.229094 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:02:08.229637 8734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.203088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54862)
0926 07:02:08.203629 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.203638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.203706 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.203861 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.203862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.204036 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.204107 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.204109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.205028 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.205029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.205856 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.205859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.206371 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.206423 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.225413 (+ 18990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.225416 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.226001 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.227929 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.227930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.227931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228198 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228389 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228389 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.228389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228444 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228730 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.228731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.228803 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.229420 (+ 617us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.229526 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":472,"threads_started":1}
I20260926 07:02:08.229763 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8053
I20260926 07:02:08.230542 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.202816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44612)
0926 07:02:08.203001 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.203010 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.203503 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.203685 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.203685 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.203859 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.203910 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.203911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.204929 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.204930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.205742 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.205743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.206251 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.206251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.206317 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.225406 (+ 19089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.225411 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.225989 (+ 578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.227992 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.227993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.227993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228263 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228466 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228466 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.228466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.228522 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.228817 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.228818 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.228853 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.229682 (+ 829us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.230454 (+ 772us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260926 07:02:08.231916 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.211994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.1.217.124:38647)
0926 07:02:08.216877 (+ 4883us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.216879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.216885 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:02:08.216913 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.216914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.216922 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.217018 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.217285 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.217761 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.218432 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.218634 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.218710 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.220746 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.220748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.220748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.220755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.227986 (+ 7231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228394 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228458 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.228649 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.228650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.228650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.228812 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.229348 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.231744 (+ 2396us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.231825 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4864,"thread_start_us":93,"threads_started":1}
I20260926 07:02:08.232323 7991 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.1.217.124:38647: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "21339434331f4ebcac499ec595ee3935"
member_type: NON_VOTER
last_known_addr {
host: "127.1.217.124"
port: 38647
}
attrs {
promote: true
}
}
dest_uuid: "b9d6220d324442efb2c4200fa924ac73"
cas_config_opid_index: 5
I20260926 07:02:08.232565 7991 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } }
I20260926 07:02:08.232904 8550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 21339434331f4ebcac499ec595ee3935 (127.1.217.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.233353 8684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:08.233865 8550 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.1.217.124:38647
I20260926 07:02:08.233930 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.233983 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.234256 8684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:02:08.234272 8642 master.cc:537] Master@127.1.217.124:38647 shutting down...
I20260926 07:02:08.235997 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:43787
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:38595
--webserver_port=38367
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:02:08.237934 8642 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:08.238026 8642 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:08.238082 8642 tablet_replica.cc:330] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: stopping tablet replica
I20260926 07:02:08.238143 8549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:08.240872 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.233047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44626)
0926 07:02:08.233240 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.233250 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.233292 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.233492 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.233493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.233494 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.233538 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.233539 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.238445 (+ 4906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.238445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.239662 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.239666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.240609 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.240610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.240679 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.240703 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.240742 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.240805 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260926 07:02:08.241007 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.239896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54878)
0926 07:02:08.240953 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.240989 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":85,"threads_started":1}
W20260926 07:02:08.241084 7978 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 -> Peer 21339434331f4ebcac499ec595ee3935 (127.1.217.124:38647): Couldn't send request to peer 21339434331f4ebcac499ec595ee3935. Status: Network error: Client connection negotiation failed: client connection to 127.1.217.124:38647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:02:08.241309 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.233130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.1.217.125:34917)
0926 07:02:08.233211 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.233213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.233339 (+ 126us) server_negotiation.cc:407] Connection header received
0926 07:02:08.233362 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.233362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.233384 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.233439 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.233610 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.234336 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.239745 (+ 5409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.240382 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.241111 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.241185 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.241205 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.241246 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 07:02:08.241539 8598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 LEARNER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 07:02:08.241730 8549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 07:02:08.242234 8598 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 21339434331f4ebcac499ec595ee3935 (127.1.217.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.243554 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d6220d324442efb2c4200fa924ac73. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.243649 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: LEARNER
I20260926 07:02:08.243688 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: NON_VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: true } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.243757 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: LEARNER
I20260926 07:02:08.244038 8549 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: attempting to promote NON_VOTER 317927315e3044b9bb73241a4e5b0d4a to VOTER
I20260926 07:02:08.244246 8549 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } }
I20260926 07:02:08.245085 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.244945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54892)
0926 07:02:08.245006 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.245047 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:02:08.245337 8598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 LEARNER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 07:02:08.245373 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.245300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54904)
0926 07:02:08.245332 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.245352 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 07:02:08.245427 7978 consensus_peers.cc:487] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 -> Peer 21339434331f4ebcac499ec595ee3935 (127.1.217.124:38647): Couldn't send request to peer 21339434331f4ebcac499ec595ee3935. Status: Network error: Client connection negotiation failed: client connection to 127.1.217.124:38647: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:02:08.245539 8549 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 07:02:08.246338 8550 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 317927315e3044b9bb73241a4e5b0d4a (127.1.217.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.246493 8598 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 317927315e3044b9bb73241a4e5b0d4a (127.1.217.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.247198 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.247267 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.247310 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.247349 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.247442 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.247545 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:02:08.251883 8642 master.cc:552] Master@127.1.217.124:38647 shutdown complete.
I20260926 07:02:08.251935 8642 master_runner.cc:305] Clearing existing system tablet
I20260926 07:02:08.252064 8642 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.252120 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.252880 8642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 07:02:08.253409 8742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.253553 8642 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.253616 8642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.253641 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "21339434331f4ebcac499ec595ee3935"
format_stamp: "Formatted at 2026-09-26 07:02:08 on dist-test-slave-t1mc"
server_key: "32971bf1490ae83f28fa19b5f5bba538"
server_key_iv: "17856875dd7c6055aa35338a4d437db3"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.254415 8642 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:08.255891 8642 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 07:02:08.255960 8642 log.cc:1196] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 07:02:08.256167 8642 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Deleting consensus metadata
I20260926 07:02:08.256325 8642 master_runner.cc:315] Copying system tablet from 127.1.217.126:38595
I20260926 07:02:08.257151 8642 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: tablet copy: Beginning tablet copy session from remote peer at address 127.1.217.126:38595
I20260926 07:02:08.264336 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.257613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.1.217.126:38595)
0926 07:02:08.257653 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.257655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.257665 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:08.257715 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.257716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.257726 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.257792 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.258144 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.258937 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.260017 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.260327 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.260391 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.261552 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.261554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.261554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.261561 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.263157 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.263544 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.263545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.263546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.263604 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.263816 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.263817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.263817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.263944 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.264010 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.264170 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.264247 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:08.264372 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.257442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:45602)
0926 07:02:08.257582 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.257595 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.257659 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.257798 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.257799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.258004 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.258056 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.258058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.259024 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.259025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.259906 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.259909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.260400 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.260401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.260461 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.261036 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.261042 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.261450 (+ 408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.263238 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.263239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.263239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.263459 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.263670 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.263671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.263671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.263743 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.264009 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.264010 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.264057 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.264136 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.264241 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 07:02:08.264914 8011 tablet_copy_service.cc:140] P b9d6220d324442efb2c4200fa924ac73: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 21339434331f4ebcac499ec595ee3935 ({username='oryx', principal='oryx/127.1.217.124@KRBTEST.COM'} at 127.0.0.1:45602)
I20260926 07:02:08.264964 8011 tablet_copy_service.cc:161] P b9d6220d324442efb2c4200fa924ac73: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 21339434331f4ebcac499ec595ee3935 at {username='oryx', principal='oryx/127.1.217.124@KRBTEST.COM'} at 127.0.0.1:45602: session id = 21339434331f4ebcac499ec595ee3935-00000000000000000000000000000000
I20260926 07:02:08.265504 8011 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 07:02:08.265966 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 07:02:08.267239 8642 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: tablet copy: Starting download of 0 data blocks...
I20260926 07:02:08.267380 8642 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: tablet copy: Starting download of 1 WAL segments...
I20260926 07:02:08.268211 8642 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 07:02:08.269124 8011 tablet_copy_service.cc:342] P b9d6220d324442efb2c4200fa924ac73: Request end of tablet copy session 21339434331f4ebcac499ec595ee3935-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.1.217.124@KRBTEST.COM'} at 127.0.0.1:45602
I20260926 07:02:08.269181 8011 tablet_copy_service.cc:433] P b9d6220d324442efb2c4200fa924ac73: ending tablet copy session 21339434331f4ebcac499ec595ee3935-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 21339434331f4ebcac499ec595ee3935
I20260926 07:02:08.271713 8642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:08.272864 8749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:08.272907 8750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.277830 8642 server_base.cc:1034] running on GCE node
W20260926 07:02:08.277910 8752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.278044 8642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:08.278206 8642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:08.279346 8642 hybrid_clock.cc:648] HybridClock initialized: now 1790406128279339 us; error 61 us; skew 500 ppm
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:08.280881 8642 init.cc:377] Logged in from keytab as oryx/127.1.217.124@KRBTEST.COM (short username oryx)
I20260926 07:02:08.281128 8642 webserver.cc:460] Webserver started at http://127.1.217.124:44713/ using document root <none> and password file <none>
I20260926 07:02:08.281242 8642 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.281275 8642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.281983 8642 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:08.282572 8759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.282688 8642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.282727 8642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "21339434331f4ebcac499ec595ee3935"
format_stamp: "Formatted at 2026-09-26 07:02:08 on dist-test-slave-t1mc"
server_key: "32971bf1490ae83f28fa19b5f5bba538"
server_key_iv: "17856875dd7c6055aa35338a4d437db3"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.282787 8642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.304046 8642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:08.305985 8642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:08.309047 8642 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.124:38647
I20260926 07:02:08.309077 8811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.124:38647 every 8 connection(s)
I20260926 07:02:08.309384 8642 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
I20260926 07:02:08.310267 8812 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:02:08.310781 8812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Bootstrap starting.
I20260926 07:02:08.313694 8812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:02:08.313923 8812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: Bootstrap complete.
I20260926 07:02:08.314136 8812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } }
I20260926 07:02:08.314230 8812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 21339434331f4ebcac499ec595ee3935, State: Initialized, Role: LEARNER
I20260926 07:02:08.314267 8812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } }
I20260926 07:02:08.314450 8812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:08.314469 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.314702 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: LEARNER
I20260926 07:02:08.316192 8826 catalog_manager.cc:1259] Loaded cluster ID: 7c9dece23a06422896704a4cc0192c29
I20260926 07:02:08.316382 8826 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: loading cluster ID for follower catalog manager: success
I20260926 07:02:08.317607 8826 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: acquiring CA information for follower catalog manager: success
I20260926 07:02:08.317912 8826 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 07:02:08.332311 8736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:08.332491 8736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:08.332512 8736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:08.333698 8736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:08.333739 8736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:08.333765 8736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:08.335191 8736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:43787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=38367
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:08.335405 8736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:08.335628 8736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:08.341451 8832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:08.341576 8831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.341595 8736 server_base.cc:1034] running on GCE node
W20260926 07:02:08.341463 8834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.341887 8736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:08.342175 8736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:08.343343 8736 hybrid_clock.cc:648] HybridClock initialized: now 1790406128343316 us; error 39 us; skew 500 ppm
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:08.346711 8736 init.cc:377] Logged in from keytab as oryx/127.1.217.65@KRBTEST.COM (short username oryx)
I20260926 07:02:08.347116 8736 webserver.cc:460] Webserver started at http://127.1.217.65:38367/ using document root <none> and password file <none>
I20260926 07:02:08.347330 8736 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.347409 8736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.348824 8736 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:08.349483 8841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.349653 8736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:08.349748 8736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "62c25104d4b141ccbe833b90edb35618"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "5280e130655f02151026225f007e1da5"
server_key_iv: "ed0aec46888bbca853e5f666ac1889d5"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.350082 8736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.373117 8736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:08.375396 8736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:08.375560 8736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:08.375782 8736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:08.376088 8736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:08.376142 8736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.376183 8736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:08.376211 8736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.382174 8736 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43787
I20260926 07:02:08.382223 8954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43787 every 8 connection(s)
I20260926 07:02:08.382650 8736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:08.388487 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8736
I20260926 07:02:08.388636 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8187
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
I20260926 07:02:08.395527 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:56461 (local address 127.1.217.125:34917)
0926 07:02:08.383496 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.383499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.383810 (+ 311us) server_negotiation.cc:407] Connection header received
0926 07:02:08.384706 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.384707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.384719 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.384805 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.386230 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.386666 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387608 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.387884 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387954 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.391722 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.391724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.391725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.391733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.394114 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394579 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394634 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394825 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394964 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.395005 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.395356 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.395434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 07:02:08.395587 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:54727 (local address 127.1.217.126:38595)
0926 07:02:08.383760 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.383761 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.384934 (+ 1173us) server_negotiation.cc:407] Connection header received
0926 07:02:08.384982 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.384983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.384995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.385045 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.385650 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.386152 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387354 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.387662 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387728 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.392167 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.392169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.392169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.392177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.393896 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394469 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394535 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394806 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394946 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.394988 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.395363 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.395453 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:02:08.395969 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:44129 (local address 127.1.217.124:38647)
0926 07:02:08.383485 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.383488 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.384728 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 07:02:08.384754 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.384755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.384767 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.384849 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.386743 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.387536 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.388475 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.389381 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.389472 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.392175 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.392176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.392176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.392183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.394021 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394516 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394577 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394816 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.394817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.394817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.394942 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.394996 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.395788 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.395873 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260926 07:02:08.396013 8960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:54727)
0926 07:02:08.384856 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.384865 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.384928 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.385064 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.385065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.385347 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.385548 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.385554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.386234 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.386235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.387273 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.387276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.387739 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.387929 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.389540 (+ 1611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.389544 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.392081 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.393986 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.393992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.393995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394383 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.394604 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.394605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394738 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.395030 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.395035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.395200 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.395661 (+ 461us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.395776 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1201,"mutex_wait_us":5,"thread_start_us":60,"threads_started":1}
I20260926 07:02:08.396270 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:43059
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:38595
--webserver_port=44537
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:02:08.396294 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.1.217.65:44129)
0926 07:02:08.383602 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.383641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.384604 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.384893 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.384894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.385356 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.385548 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.385553 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.387626 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.387627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.388389 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.388392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.389472 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.389473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.389561 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.389944 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.389949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.392081 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.394117 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.394118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394427 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.394647 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394647 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.394648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394744 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.395031 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.395037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.395201 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.395313 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.395483 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260926 07:02:08.396469 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.383332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.1.217.65:56461)
0926 07:02:08.383746 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.383756 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.384620 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.384886 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.384890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.385352 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.385548 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.385554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.386742 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.386743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.387524 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.387525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.388007 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.388008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.388073 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.388667 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.388679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.391623 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.394191 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.394193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394498 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.394706 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.394706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.394707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.394753 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.395050 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.395050 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.395201 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.395836 (+ 635us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.395926 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260926 07:02:08.396874 8957 heartbeater.cc:347] Connected to a master server at 127.1.217.125:34917
I20260926 07:02:08.396997 8957 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.397199 8957 heartbeater.cc:511] Master 127.1.217.125:34917 requested a full tablet report, sending...
I20260926 07:02:08.397362 8956 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:08.397403 8956 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.397470 8956 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:08.397699 8588 ts_manager.cc:194] Registered new tserver with Master: 62c25104d4b141ccbe833b90edb35618 (127.1.217.65:43787)
I20260926 07:02:08.397698 7991 ts_manager.cc:194] Re-registered known tserver with Master: 62c25104d4b141ccbe833b90edb35618 (127.1.217.65:43787)
I20260926 07:02:08.397914 8955 heartbeater.cc:347] Connected to a master server at 127.1.217.124:38647
I20260926 07:02:08.397975 8955 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.398042 8955 heartbeater.cc:511] Master 127.1.217.124:38647 requested a full tablet report, sending...
I20260926 07:02:08.398191 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:54727
I20260926 07:02:08.398409 8770 ts_manager.cc:194] Registered new tserver with Master: 62c25104d4b141ccbe833b90edb35618 (127.1.217.65:43787)
W20260926 07:02:08.477787 8963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:08.477988 8963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:08.478010 8963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:08.479178 8963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:08.479220 8963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:08.479246 8963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:08.480713 8963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:43059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=44537
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:08.480978 8963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:08.481218 8963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:08.486207 8969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:08.486218 8971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.486423 8963 server_base.cc:1034] running on GCE node
W20260926 07:02:08.486527 8968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.486744 8963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:08.486989 8963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:08.488186 8963 hybrid_clock.cc:648] HybridClock initialized: now 1790406128488173 us; error 29 us; skew 500 ppm
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:08.490630 8963 init.cc:377] Logged in from keytab as oryx/127.1.217.66@KRBTEST.COM (short username oryx)
I20260926 07:02:08.491066 8963 webserver.cc:460] Webserver started at http://127.1.217.66:44537/ using document root <none> and password file <none>
I20260926 07:02:08.491248 8963 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.491297 8963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.492542 8963 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:08.493120 8978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.493299 8963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.493402 8963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "c8acfe0b9053c6cf83eace12976dd43c"
server_key_iv: "4e15c7b34305fb79f374ef1ad8f9ab6b"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.493764 8963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.517143 8963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:08.519104 8963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:08.519263 8963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:08.519505 8963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:08.519842 8963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:08.519896 8963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.519934 8963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:08.519960 8963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.525799 8963 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:43059
I20260926 07:02:08.525847 9091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:43059 every 8 connection(s)
I20260926 07:02:08.526167 8963 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:08.531356 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 8963
I20260926 07:02:08.531526 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8321
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
I20260926 07:02:08.537465 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:37337
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:38595
--webserver_port=37381
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:02:08.538033 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.527006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:51685 (local address 127.1.217.126:38595)
0926 07:02:08.527052 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.527054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.527257 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:02:08.527858 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.527860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.527873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.527913 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.528571 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529364 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530959 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.531208 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.531272 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.534125 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.534126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.534127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.534134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.535835 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.536252 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.536253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.536254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.536310 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.537225 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.537226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.537226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.537375 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.537462 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.537867 (+ 405us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.537938 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:02:08.538035 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.527023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:44459 (local address 127.1.217.124:38647)
0926 07:02:08.527071 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.527073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.527373 (+ 300us) server_negotiation.cc:407] Connection header received
0926 07:02:08.527864 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.527866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.527880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.527911 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.528636 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529383 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530423 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.530704 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530772 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.534612 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.534614 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.534614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.534622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.536304 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.536628 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.536629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.536630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.536689 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.537300 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.537301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.537302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.537458 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.537522 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.537838 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.537945 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
I20260926 07:02:08.538623 9098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.526869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.1.217.66:44459)
0926 07:02:08.527308 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.527319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.527771 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.528008 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.528012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.528308 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.528481 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.528487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529470 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.529471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.530323 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.530325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.530775 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.530868 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.533200 (+ 2332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.533206 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.534532 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.536382 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.536383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.536384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.536537 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.536763 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.536788 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.536788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.537219 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.537548 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.537549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.537684 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.538205 (+ 521us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.538389 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":121,"thread_start_us":74,"threads_started":1}
I20260926 07:02:08.538628 9097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.526870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:51685)
0926 07:02:08.527122 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.527162 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.527749 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.528013 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.528014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.528285 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.528478 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.528482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529444 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.529444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.530487 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.530490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.531286 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.531287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.531368 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.532172 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.532177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.534014 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.535932 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.535939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.535941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.536161 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.536620 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.536621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.536621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.536770 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.537470 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.537476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.537744 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.538461 (+ 717us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.538546 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":76,"thread_start_us":91,"threads_started":1}
I20260926 07:02:08.539309 9093 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:08.539414 9093 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.539583 9092 heartbeater.cc:347] Connected to a master server at 127.1.217.124:38647
I20260926 07:02:08.539626 9092 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.539633 9093 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:08.539674 9092 heartbeater.cc:511] Master 127.1.217.124:38647 requested a full tablet report, sending...
I20260926 07:02:08.540061 7991 ts_manager.cc:194] Re-registered known tserver with Master: 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:08.540072 8770 ts_manager.cc:194] Registered new tserver with Master: 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:08.540585 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:51685
I20260926 07:02:08.541479 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.527179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.1.217.66:57851)
0926 07:02:08.527327 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.527337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.527770 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.528387 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.528387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.528535 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.528592 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.528594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529307 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.529308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.530487 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.530490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.530929 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.530985 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.532007 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.532019 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.538583 (+ 6564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.540492 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.540493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.540493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.540659 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.540895 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.540895 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.540896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.540953 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.541226 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.541226 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.541258 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.541310 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.541389 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":104,"thread_start_us":77,"threads_started":1}
I20260926 07:02:08.541572 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.527293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:57851 (local address 127.1.217.125:34917)
0926 07:02:08.527331 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.527333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.527384 (+ 51us) server_negotiation.cc:407] Connection header received
0926 07:02:08.527864 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.527865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.527878 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.527911 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.528653 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.529200 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530578 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.530863 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.530910 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.538699 (+ 7789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.538701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.538702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.538711 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.540400 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.540747 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.540749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.540749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.540825 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.541023 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.541024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.541025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.541159 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.541232 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.541336 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.541424 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:08.541963 9094 heartbeater.cc:347] Connected to a master server at 127.1.217.125:34917
I20260926 07:02:08.542017 9094 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.542079 9094 heartbeater.cc:511] Master 127.1.217.125:34917 requested a full tablet report, sending...
I20260926 07:02:08.542357 8588 ts_manager.cc:194] Registered new tserver with Master: 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
W20260926 07:02:08.621220 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:08.621418 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:08.621441 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:08.622512 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:08.622546 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:08.622572 9099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:08.624010 9099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:37337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=37381
--tserver_master_addrs=127.1.217.126:38595,127.1.217.125:34917,127.1.217.124:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:08.624192 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:08.624375 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:08.629263 9105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.629279 9099 server_base.cc:1034] running on GCE node
W20260926 07:02:08.629249 9104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:08.629233 9107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:08.629722 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:08.629920 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:08.631063 9099 hybrid_clock.cc:648] HybridClock initialized: now 1790406128631052 us; error 28 us; skew 500 ppm
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:08.633600 9099 init.cc:377] Logged in from keytab as oryx/127.1.217.67@KRBTEST.COM (short username oryx)
I20260926 07:02:08.634068 9099 webserver.cc:460] Webserver started at http://127.1.217.67:37381/ using document root <none> and password file <none>
I20260926 07:02:08.634284 9099 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:08.634338 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:08.635778 9099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:08.636310 9114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.636467 9099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:08.636565 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
format_stamp: "Formatted at 2026-09-26 07:02:07 on dist-test-slave-t1mc"
server_key: "ddb801c7a3c0cfd2ed6f60e01530b2a1"
server_key_iv: "c0a69e006251c813ed64057398c69d62"
server_key_version: "encryptionkey@0"
I20260926 07:02:08.636952 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:08.644444 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:08.646670 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:08.646824 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:08.647032 9099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:08.647322 9099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:08.647375 9099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.647416 9099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:08.647444 9099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:08.653272 9099 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:37337
I20260926 07:02:08.653335 9227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:37337 every 8 connection(s)
I20260926 07:02:08.653664 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:08.654284 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 9099
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.125@KRBTEST.COM
Sep 26 07:02:08 dist-test-slave-t1mc krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406128, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.124@KRBTEST.COM
I20260926 07:02:08.671831 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.656005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44632 (local address 127.1.217.125:34917)
0926 07:02:08.656186 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.656188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.660414 (+ 4226us) server_negotiation.cc:407] Connection header received
0926 07:02:08.660436 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.660437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.660450 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.660510 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.661239 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.661935 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.662840 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.663024 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.663078 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.666943 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.666945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.666945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.666953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.670030 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.670424 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.670426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.670426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.670485 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.670702 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.670704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.670704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.670880 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.671163 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.671561 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.671645 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":57,"threads_started":1}
I20260926 07:02:08.673334 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:58737 (local address 127.1.217.124:38647)
0926 07:02:08.654909 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.654912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.654931 (+ 19us) server_negotiation.cc:407] Connection header received
0926 07:02:08.655589 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.655591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.655604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.655636 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.656595 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657380 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659074 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659373 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659416 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.669874 (+ 10458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.669875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.669876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.669883 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.671627 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672138 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.672139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.672139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.672195 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672479 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.672480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.672481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.672619 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.672682 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.673118 (+ 436us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.673203 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513}
I20260926 07:02:08.673419 8049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:42779 (local address 127.1.217.126:38595)
0926 07:02:08.654870 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.654872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.655075 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:02:08.655763 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.655764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.655776 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.655828 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.656576 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657363 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659087 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659393 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659449 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.669533 (+ 10084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.669535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.669535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.669543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.671841 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672308 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.672309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.672309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.672359 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672578 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.672579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.672579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.672710 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.672801 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.673246 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.673326 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 07:02:08.673780 9232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:42779)
0926 07:02:08.654995 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.655007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.655373 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.655881 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.655882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.656237 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.656491 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.656497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657470 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.657471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.658967 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.658969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659496 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.659589 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.665981 (+ 6392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.665992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.669451 (+ 3459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.672027 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.672029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.672227 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.672429 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.672431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.672492 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.672830 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.672831 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.672958 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.673068 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.673395 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":31,"thread_start_us":47,"threads_started":1}
I20260926 07:02:08.674109 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.655658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.1.217.124:38647)
0926 07:02:08.656240 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.656242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.656249 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:08.656269 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.656270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.656281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.656344 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.660403 (+ 4059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.661246 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.662147 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.662326 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.662374 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.668225 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.668226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.668226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.668234 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.672621 (+ 4387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.673034 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.673035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.673035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.673092 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.673279 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.673279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.673280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.673420 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.673509 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.673917 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.674002 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":61,"threads_started":1}
I20260926 07:02:08.674294 9229 heartbeater.cc:347] Connected to a master server at 127.1.217.126:38595
I20260926 07:02:08.674389 9229 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.674608 9229 heartbeater.cc:511] Master 127.1.217.126:38595 requested a full tablet report, sending...
I20260926 07:02:08.674752 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:33495 (local address 127.1.217.125:34917)
0926 07:02:08.654701 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.654703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.654935 (+ 232us) server_negotiation.cc:407] Connection header received
0926 07:02:08.655510 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.655512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.655524 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.655558 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.656849 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657337 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.658958 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659291 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659338 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.669563 (+ 10225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.669564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.669564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.669571 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.672880 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.673917 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.673918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.673919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.673978 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.674238 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.674239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.674239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.674356 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.674397 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.674601 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.674678 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:08.674971 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.1.217.67:33495)
0926 07:02:08.654769 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.654885 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.655375 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.656115 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.656116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.656692 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.656747 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.656748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657639 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.657640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.658857 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.658860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659446 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.659582 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.665995 (+ 6413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.666000 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.669451 (+ 3451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.673085 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.673086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.673086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.673291 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.674077 (+ 786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.674078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.674078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.674134 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.674443 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.674444 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.674481 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.674549 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.674634 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
I20260926 07:02:08.675014 7991 ts_manager.cc:194] Re-registered known tserver with Master: daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337)
I20260926 07:02:08.673786 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.654383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.1.217.67:58737)
0926 07:02:08.654657 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.654701 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.655499 (+ 798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.655720 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.655725 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.656182 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:08.656492 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.656500 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.657472 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.657472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.658963 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.658966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.659497 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.659497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.659589 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.666252 (+ 6663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:08.666257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:08.669748 (+ 3491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:08.671719 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.671726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.671728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.672024 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.672268 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:08.672269 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:08.672269 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:08.672426 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:08.672710 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:08.672717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:08.672959 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.673410 (+ 451us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.673581 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":34,"thread_start_us":81,"threads_started":1}
I20260926 07:02:08.675396 9239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.655452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.1.217.126:38595)
0926 07:02:08.656387 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.656389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.656396 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:08.656418 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.656418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.656444 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.656504 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:08.659406 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.660048 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.661070 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.661305 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.661368 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.667037 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:08.667038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:08.667039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:08.667047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:08.668984 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.670912 (+ 1928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.670913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.670913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.670978 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:08.671191 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:08.671192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:08.671192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:08.671333 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:08.675026 (+ 3693us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.675088 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.675297 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":908,"thread_start_us":78,"threads_started":1}
I20260926 07:02:08.675540 7991 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:42779
I20260926 07:02:08.675858 9228 heartbeater.cc:347] Connected to a master server at 127.1.217.124:38647
I20260926 07:02:08.675858 9230 heartbeater.cc:347] Connected to a master server at 127.1.217.125:34917
I20260926 07:02:08.675923 9228 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.675935 9230 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:08.676016 9228 heartbeater.cc:511] Master 127.1.217.124:38647 requested a full tablet report, sending...
I20260926 07:02:08.676016 9230 heartbeater.cc:511] Master 127.1.217.125:34917 requested a full tablet report, sending...
I20260926 07:02:08.676445 8770 ts_manager.cc:194] Registered new tserver with Master: daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337)
I20260926 07:02:08.676693 8588 ts_manager.cc:194] Registered new tserver with Master: daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337)
I20260926 07:02:08.677213 7990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:08.678711 7990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:08.686311 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43787 (local address 127.0.0.1:47974)
0926 07:02:08.682331 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.682341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.682388 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.682748 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.682749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.682751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.682807 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.682809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.684139 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.684140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.685255 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.685256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.686147 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.686147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.686190 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.686195 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.686228 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.686258 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260926 07:02:08.686357 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47974 (local address 127.1.217.65:43787)
0926 07:02:08.682537 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.682543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.682564 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:02:08.682611 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.682613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.682629 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.682747 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.682895 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.684060 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.685335 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.686070 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.686143 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.686217 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.686244 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.686277 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260926 07:02:08.686725 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43059 (local address 127.0.0.1:40510)
0926 07:02:08.682748 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.682758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.682807 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.683078 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.683079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.683081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.683118 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.683120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.684592 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.684593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.685614 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.685617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.686519 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.686520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.686579 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.686583 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.686612 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.686646 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":101,"threads_started":1}
I20260926 07:02:08.687125 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.1.217.66:43059)
0926 07:02:08.682915 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.682920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.682928 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:08.682966 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.682968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.682982 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.683189 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.683290 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.684477 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.685715 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.686421 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.686893 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.686975 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.687001 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.687042 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":75,"threads_started":1}
I20260926 07:02:08.687088 8889 tablet_service.cc:1467] Processing CreateTablet for tablet e1ab95dd5a80491f8e7a39880883c662 (DEFAULT_TABLE table=test-table [id=151bc25146684363b9e9da8bfefe3787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:08.687459 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ab95dd5a80491f8e7a39880883c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:08.687785 9026 tablet_service.cc:1467] Processing CreateTablet for tablet e1ab95dd5a80491f8e7a39880883c662 (DEFAULT_TABLE table=test-table [id=151bc25146684363b9e9da8bfefe3787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:08.688087 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ab95dd5a80491f8e7a39880883c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:08.689105 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37337 (local address 127.0.0.1:37872)
0926 07:02:08.682385 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.682395 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.682424 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.683044 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.683045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.683047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.683083 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.683085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.684017 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.684018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.685248 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.685251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.688859 (+ 3608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.688860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.688944 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.688948 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.689000 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.689044 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":43}
I20260926 07:02:08.689144 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.682496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.1.217.67:37337)
0926 07:02:08.682896 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.682901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.682909 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:08.682941 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.682942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.682960 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.683047 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.683136 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.683933 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.686476 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.687489 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.687543 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.687649 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.689028 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.689069 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":98,"threads_started":1}
I20260926 07:02:08.691553 9253 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Bootstrap starting.
I20260926 07:02:08.691915 9254 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Bootstrap starting.
I20260926 07:02:08.692157 9162 tablet_service.cc:1467] Processing CreateTablet for tablet e1ab95dd5a80491f8e7a39880883c662 (DEFAULT_TABLE table=test-table [id=151bc25146684363b9e9da8bfefe3787]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:08.692268 9253 tablet_bootstrap.cc:654] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:08.692441 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1ab95dd5a80491f8e7a39880883c662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:08.693040 9253 log.cc:824] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:08.693102 9254 tablet_bootstrap.cc:654] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:08.693564 9254 log.cc:824] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:08.693853 9253 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: No bootstrap required, opened a new log
I20260926 07:02:08.693949 9253 ts_tablet_manager.cc:1397] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:02:08.694295 9254 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: No bootstrap required, opened a new log
I20260926 07:02:08.694366 9254 ts_tablet_manager.cc:1397] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:02:08.695318 9253 raft_consensus.cc:348] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.695479 9253 raft_consensus.cc:374] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:08.695523 9253 raft_consensus.cc:729] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c25104d4b141ccbe833b90edb35618, State: Initialized, Role: FOLLOWER
I20260926 07:02:08.695648 9253 consensus_queue.cc:260] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.695838 9257 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Bootstrap starting.
I20260926 07:02:08.695909 9253 ts_tablet_manager.cc:1428] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:02:08.696025 8956 heartbeater.cc:503] Master 127.1.217.126:38595 was elected leader, sending a full tablet report...
I20260926 07:02:08.696362 9254 raft_consensus.cc:348] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.696506 9254 raft_consensus.cc:374] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:08.696538 9254 raft_consensus.cc:729] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b11c6a9dffc4b04a7e929cc980b2303, State: Initialized, Role: FOLLOWER
I20260926 07:02:08.696635 9254 consensus_queue.cc:260] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.696924 9254 ts_tablet_manager.cc:1428] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 07:02:08.697026 9093 heartbeater.cc:503] Master 127.1.217.126:38595 was elected leader, sending a full tablet report...
I20260926 07:02:08.697264 9257 tablet_bootstrap.cc:654] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:08.697597 9257 log.cc:824] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:08.698307 9257 tablet_bootstrap.cc:492] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: No bootstrap required, opened a new log
I20260926 07:02:08.698427 9257 ts_tablet_manager.cc:1397] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:02:08.699719 9257 raft_consensus.cc:348] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.699834 9257 raft_consensus.cc:374] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:08.699855 9257 raft_consensus.cc:729] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf3b140bf7349e8bd8cd78ce8f933fc, State: Initialized, Role: FOLLOWER
I20260926 07:02:08.699930 9257 consensus_queue.cc:260] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.700098 9257 ts_tablet_manager.cc:1428] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:02:08.700176 9229 heartbeater.cc:503] Master 127.1.217.126:38595 was elected leader, sending a full tablet report...
W20260926 07:02:08.776966 9095 tablet.cc:2369] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:08.799034 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.796294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54928)
0926 07:02:08.796344 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.796357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.796429 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.796661 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.796662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.796664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.796713 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.796714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.797517 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.797517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.798209 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.798211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.798813 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.798814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.798871 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.798874 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.798935 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.798975 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:02:08.799032 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.796460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.1.217.124:38647)
0926 07:02:08.796523 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.796525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.796534 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:08.796568 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.796569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.796590 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.796648 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.796856 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.797440 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.798290 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.798715 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.798778 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.798834 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.798942 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.798974 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:02:08.799516 8786 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 07:02:08.800395 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9d6220d324442efb2c4200fa924ac73. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.800508 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: LEARNER
W20260926 07:02:08.885439 8958 tablet.cc:2369] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:02:08.904485 9231 tablet.cc:2369] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:08.955402 9258 raft_consensus.cc:482] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:08.955565 9258 raft_consensus.cc:504] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.955974 9258 leader_election.cc:288] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337), 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:08.958963 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.956367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:43649 (local address 127.1.217.67:37337)
0926 07:02:08.956412 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.956414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.956447 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:02:08.956536 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.956537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.956560 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.956616 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.956806 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.957279 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958283 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.958624 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958678 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.958706 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.958846 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.958890 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:02:08.958981 9267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.956222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37337 (local address 127.1.217.65:43649)
0926 07:02:08.956382 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.956392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.956465 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.956637 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.956638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.956640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.956705 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.956707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.957368 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.957369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.958174 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.958175 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.958705 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.958768 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.958776 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.958818 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.958850 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260926 07:02:08.959016 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.956298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:33735 (local address 127.1.217.66:43059)
0926 07:02:08.956371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:08.956374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:08.956450 (+ 76us) server_negotiation.cc:407] Connection header received
0926 07:02:08.956500 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:08.956501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:08.956511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:08.956573 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:08.956719 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.957227 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958270 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.958631 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958718 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.958726 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:08.958902 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:02:08.958946 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:02:08.959256 9266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:08.956207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43059 (local address 127.1.217.65:33735)
0926 07:02:08.956358 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:08.956370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:08.956442 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:08.956585 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:08.956586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:08.956589 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:08.956647 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.956650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.957308 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.957309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.958174 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:08.958176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:08.958735 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:08.958736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:08.958819 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:08.958823 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:08.958879 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:02:08.958913 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":34,"thread_start_us":77,"threads_started":1}
I20260926 07:02:08.959411 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ab95dd5a80491f8e7a39880883c662" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" is_pre_election: true
I20260926 07:02:08.959583 9182 raft_consensus.cc:2454] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 0.
I20260926 07:02:08.959834 8842 leader_election.cc:302] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c25104d4b141ccbe833b90edb35618, daf3b140bf7349e8bd8cd78ce8f933fc; no voters:
I20260926 07:02:08.959829 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ab95dd5a80491f8e7a39880883c662" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" is_pre_election: true
I20260926 07:02:08.959968 9046 raft_consensus.cc:2454] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 0.
I20260926 07:02:08.960012 9258 raft_consensus.cc:2781] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:08.960086 9258 raft_consensus.cc:482] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:08.960116 9258 raft_consensus.cc:3037] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:08.960822 9258 raft_consensus.cc:504] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.961041 9258 leader_election.cc:288] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 election: Requested vote from peers daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337), 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:08.961133 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ab95dd5a80491f8e7a39880883c662" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
I20260926 07:02:08.961159 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1ab95dd5a80491f8e7a39880883c662" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
I20260926 07:02:08.961198 9046 raft_consensus.cc:3037] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:08.961238 9182 raft_consensus.cc:3037] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:08.961830 9046 raft_consensus.cc:2454] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 1.
I20260926 07:02:08.962018 8843 leader_election.cc:302] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b11c6a9dffc4b04a7e929cc980b2303, 62c25104d4b141ccbe833b90edb35618; no voters:
I20260926 07:02:08.962581 9258 raft_consensus.cc:2781] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:08.962713 9258 raft_consensus.cc:686] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 1 LEADER]: Becoming Leader. State: Replica: 62c25104d4b141ccbe833b90edb35618, State: Running, Role: LEADER
I20260926 07:02:08.962843 9258 consensus_queue.cc:237] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } }
I20260926 07:02:08.962937 9182 raft_consensus.cc:2454] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 1.
I20260926 07:02:08.963653 7990 catalog_manager.cc:5495] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c25104d4b141ccbe833b90edb35618 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "62c25104d4b141ccbe833b90edb35618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } health_report { overall_health: UNKNOWN } } }
I20260926 07:02:08.964221 8549 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: NON_VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: true } } }
I20260926 07:02:08.964334 8549 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.964882 8740 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: attempting to promote NON_VOTER 21339434331f4ebcac499ec595ee3935 to VOTER
I20260926 07:02:08.965003 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } }
I20260926 07:02:08.965338 8786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 LEARNER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 07:02:08.965553 8550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 07:02:08.965657 8598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Refusing update from remote peer b9d6220d324442efb2c4200fa924ac73: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 07:02:08.965829 8550 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260926 07:02:08.966210 8740 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 21339434331f4ebcac499ec595ee3935 (127.1.217.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.966352 8786 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 21339434331f4ebcac499ec595ee3935 (127.1.217.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.967022 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.967098 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.967069 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.967152 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.967154 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.967219 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:08.967211 8815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.967284 8815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:02:08.967478 8598 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 21339434331f4ebcac499ec595ee3935 (127.1.217.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.968189 9271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b9d6220d324442efb2c4200fa924ac73" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:08.968305 9271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:02:09.037038 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:09.034556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.1.217.65:43787)
0926 07:02:09.034607 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:09.034609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:09.034620 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:02:09.034662 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:09.034662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:09.034683 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:09.034765 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:09.034921 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.035403 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.036195 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.036406 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.036484 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:09.036671 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:09.036803 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:09.036910 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:09.036929 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 07:02:09.036948 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:02:09.038879 9046 raft_consensus.cc:1261] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Refusing update from remote peer 62c25104d4b141ccbe833b90edb35618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:09.038879 9182 raft_consensus.cc:1261] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Refusing update from remote peer 62c25104d4b141ccbe833b90edb35618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:09.039371 9258 consensus_queue.cc:1035] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:09.039539 9268 consensus_queue.cc:1035] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:09.043012 9273 mvcc.cc:205] Tried to move back new op lower bound from 7333503504540549120 to 7333503504232579072. Current Snapshot: MvccSnapshot[applied={T|T < 7333503504540549120}]
I20260926 07:02:09.043818 9274 mvcc.cc:205] Tried to move back new op lower bound from 7333503504540549120 to 7333503504232579072. Current Snapshot: MvccSnapshot[applied={T|T < 7333503504540549120}]
I20260926 07:02:09.044454 9275 mvcc.cc:205] Tried to move back new op lower bound from 7333503504540549120 to 7333503504232579072. Current Snapshot: MvccSnapshot[applied={T|T < 7333503504540549120}]
I20260926 07:02:09.045971 7991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:09.046082 7991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:09.047362 7991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9d6220d324442efb2c4200fa924ac73: Sending DeleteTablet for 3 replicas of tablet e1ab95dd5a80491f8e7a39880883c662
I20260926 07:02:09.047657 9026 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ab95dd5a80491f8e7a39880883c662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:09 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:40510
I20260926 07:02:09.047767 9162 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ab95dd5a80491f8e7a39880883c662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:09 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:37872
I20260926 07:02:09.047825 8889 tablet_service.cc:1514] Processing DeleteTablet for tablet e1ab95dd5a80491f8e7a39880883c662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:09 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:47974
I20260926 07:02:09.048125 9285 tablet_replica.cc:330] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: stopping tablet replica
I20260926 07:02:09.048137 9284 tablet_replica.cc:330] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: stopping tablet replica
I20260926 07:02:09.048228 9284 raft_consensus.cc:2229] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:09.048326 9285 raft_consensus.cc:2229] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:09.048343 9284 raft_consensus.cc:2258] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:09.048465 9283 tablet_replica.cc:330] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: stopping tablet replica
I20260926 07:02:09.048553 9283 raft_consensus.cc:2229] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:09.048646 9283 raft_consensus.cc:2258] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:09.048977 9284 ts_tablet_manager.cc:1905] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:09.049050 9283 ts_tablet_manager.cc:1905] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:09.049309 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 7959
I20260926 07:02:09.050813 9283 ts_tablet_manager.cc:1918] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:09.050890 9283 log.cc:1196] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/e1ab95dd5a80491f8e7a39880883c662
I20260926 07:02:09.051047 9285 raft_consensus.cc:2258] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:09.051178 9283 ts_tablet_manager.cc:1939] T e1ab95dd5a80491f8e7a39880883c662 P 4b11c6a9dffc4b04a7e929cc980b2303: Deleting consensus metadata
I20260926 07:02:09.051453 9284 ts_tablet_manager.cc:1918] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:09.051558 9285 ts_tablet_manager.cc:1905] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:09.051651 9284 log.cc:1196] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/e1ab95dd5a80491f8e7a39880883c662
I20260926 07:02:09.051932 9284 ts_tablet_manager.cc:1939] T e1ab95dd5a80491f8e7a39880883c662 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting consensus metadata
I20260926 07:02:09.053042 9285 ts_tablet_manager.cc:1918] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:09.053153 9285 log.cc:1196] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/e1ab95dd5a80491f8e7a39880883c662
I20260926 07:02:09.053539 9285 ts_tablet_manager.cc:1939] T e1ab95dd5a80491f8e7a39880883c662 P 62c25104d4b141ccbe833b90edb35618: Deleting consensus metadata
W20260926 07:02:09.055905 8979 connection.cc:535] server connection from 127.0.0.1:40510 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:02:09.055954 9115 connection.cc:535] server connection from 127.0.0.1:37872 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 07:02:09.056142 8844 connection.cc:535] server connection from 127.0.0.1:47974 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:02:09.065470 9242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:09.048754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54930 (local address 127.1.217.124:38647)
0926 07:02:09.048836 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:09.048839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:09.056753 (+ 7914us) server_negotiation.cc:407] Connection header received
0926 07:02:09.056879 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:09.056879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:09.056891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:09.056931 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:09.057756 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.058558 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.059346 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.059667 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.059731 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:09.061849 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:09.061850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:09.061851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:09.061858 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:09.063694 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.064108 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.064109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.064109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.064166 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.064396 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.064397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.064397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.064546 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:09.064625 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:09.064895 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 07:02:09.065346 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:02:09.065688 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:09.057921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44636 (local address 127.1.217.125:34917)
0926 07:02:09.057960 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:09.057962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:09.057971 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:09.057993 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:09.057994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:09.058003 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:09.058063 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:09.058729 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.059515 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.060555 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.060886 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.060926 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:09.062392 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:09.062394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:09.062394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:09.062401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:09.064477 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.064877 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.064878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.064879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.064930 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.065118 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.065119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.065119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.065236 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:09.065300 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:09.065507 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:02:09.065593 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:09.068813 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:09.058216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.1.217.124:38647)
0926 07:02:09.058253 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:09.058255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:09.059447 (+ 1192us) server_negotiation.cc:407] Connection header received
0926 07:02:09.059527 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:09.059527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:09.059538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:09.059566 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:09.060681 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.061460 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.062491 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:09.062777 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:09.062810 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:09.064153 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:09.064154 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:09.064154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:09.064159 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:09.065977 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.067830 (+ 1853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.067831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.067831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.067901 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:09.068123 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:09.068124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:09.068124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:09.068266 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:09.068371 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:09.068564 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 07:02:09.068663 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:10.044267 9293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.043965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:55127)
0926 07:02:10.044091 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.044173 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
W20260926 07:02:10.044683 8956 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:38595 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.044714 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.044297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:53827)
0926 07:02:10.044596 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.044670 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":51,"threads_started":1}
W20260926 07:02:10.044917 9229 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:38595 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.046162 9295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.045989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:34493)
0926 07:02:10.046083 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.046129 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":40,"threads_started":1}
W20260926 07:02:10.046383 9093 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:38595 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.547627 9296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d6220d324442efb2c4200fa924ac73)
I20260926 07:02:10.547737 9296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } }
I20260926 07:02:10.547931 9296 leader_election.cc:288] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595), 21339434331f4ebcac499ec595ee3935 (127.1.217.124:38647)
I20260926 07:02:10.548272 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.548055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46052)
0926 07:02:10.548169 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.548223 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260926 07:02:10.548384 9299 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b9d6220d324442efb2c4200fa924ac73)
W20260926 07:02:10.548420 8572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.548502 9299 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } }
I20260926 07:02:10.548754 9299 leader_election.cc:288] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595), 317927315e3044b9bb73241a4e5b0d4a (127.1.217.125:34917)
I20260926 07:02:10.548961 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.548800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46066)
0926 07:02:10.548891 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.548914 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 07:02:10.549072 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.548868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46074)
0926 07:02:10.548974 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.549041 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
W20260926 07:02:10.549105 8572 leader_election.cc:334] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595): Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
W20260926 07:02:10.549309 8760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.549743 9303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.549618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46082)
0926 07:02:10.549674 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.549715 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:02:10.549872 8760 leader_election.cc:334] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595): Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.551602 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.548220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.1.217.124:38647)
0926 07:02:10.548301 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.548302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.548310 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:10.548394 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.548395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.548404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.548643 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:10.548688 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.549515 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.550914 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.551280 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.551335 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.551344 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.551498 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.551539 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"thread_start_us":36,"threads_started":1}
I20260926 07:02:10.551604 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.548088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:38647 (local address 127.0.0.1:54948)
0926 07:02:10.548230 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.548240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.548304 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.548500 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:10.548501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:10.548504 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:10.548572 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.548575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.549602 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.549603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.550808 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.550811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.551359 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.551360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.551428 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.551432 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:10.551491 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 07:02:10.551538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":3,"thread_start_us":33,"threads_started":1}
I20260926 07:02:10.551859 8786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "317927315e3044b9bb73241a4e5b0d4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "21339434331f4ebcac499ec595ee3935" is_pre_election: true
I20260926 07:02:10.551959 8786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 317927315e3044b9bb73241a4e5b0d4a in term 1.
I20260926 07:02:10.552141 8575 leader_election.cc:302] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21339434331f4ebcac499ec595ee3935, 317927315e3044b9bb73241a4e5b0d4a; no voters: b9d6220d324442efb2c4200fa924ac73
I20260926 07:02:10.552295 9296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 07:02:10.552336 9296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Starting leader election (detected failure of leader b9d6220d324442efb2c4200fa924ac73)
I20260926 07:02:10.552369 9296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:02:10.552373 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.549005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34917 (local address 127.0.0.1:44648)
0926 07:02:10.549127 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.549136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.549207 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.549472 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:10.549473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:10.549475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:10.549541 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.549543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.550375 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.550376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.551351 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.551353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.552118 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.552120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.552187 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.552193 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:02:10.552255 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 07:02:10.552295 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":46,"threads_started":1}
I20260926 07:02:10.552434 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.549117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44648 (local address 127.1.217.125:34917)
0926 07:02:10.549343 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.549345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.549352 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:10.549389 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.549390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.549399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.549468 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:10.549622 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.550292 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.551430 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.552049 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.552117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.552127 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.552268 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.552308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":198,"thread_start_us":123,"threads_started":1}
I20260926 07:02:10.552717 8598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21339434331f4ebcac499ec595ee3935" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "317927315e3044b9bb73241a4e5b0d4a" is_pre_election: true
I20260926 07:02:10.553292 9296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } }
I20260926 07:02:10.553429 8598 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21339434331f4ebcac499ec595ee3935 in current term 2: Already voted for candidate 317927315e3044b9bb73241a4e5b0d4a in this term.
I20260926 07:02:10.553443 9296 leader_election.cc:288] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 election: Requested vote from peers b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595), 21339434331f4ebcac499ec595ee3935 (127.1.217.124:38647)
I20260926 07:02:10.553581 8786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "317927315e3044b9bb73241a4e5b0d4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "21339434331f4ebcac499ec595ee3935"
I20260926 07:02:10.553617 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.553511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46098)
0926 07:02:10.553566 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.553592 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:02:10.553668 8786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:02:10.553666 8760 leader_election.cc:302] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21339434331f4ebcac499ec595ee3935; no voters: 317927315e3044b9bb73241a4e5b0d4a, b9d6220d324442efb2c4200fa924ac73
I20260926 07:02:10.553920 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.553849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46102)
0926 07:02:10.553879 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.553901 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 07:02:10.554009 8572 leader_election.cc:334] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595): Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
I20260926 07:02:10.554430 8786 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 317927315e3044b9bb73241a4e5b0d4a in term 2.
I20260926 07:02:10.554503 9299 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:02:10.554579 8575 leader_election.cc:302] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21339434331f4ebcac499ec595ee3935, 317927315e3044b9bb73241a4e5b0d4a; no voters: b9d6220d324442efb2c4200fa924ac73
I20260926 07:02:10.554652 9296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:02:10.554806 9296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [term 2 LEADER]: Becoming Leader. State: Replica: 317927315e3044b9bb73241a4e5b0d4a, State: Running, Role: LEADER
I20260926 07:02:10.554893 9296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } }
I20260926 07:02:10.555227 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317927315e3044b9bb73241a4e5b0d4a. Latest consensus state: current_term: 2 leader_uuid: "317927315e3044b9bb73241a4e5b0d4a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:10.555322 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:10.555511 9308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:10.555869 9308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:10.555923 9308 catalog_manager.cc:1259] Loaded cluster ID: 7c9dece23a06422896704a4cc0192c29
I20260926 07:02:10.555939 9308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:10.556056 9308 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:10.556106 9308 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Loaded TSK: 0
I20260926 07:02:10.556191 9308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:10.601367 8587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:10.602886 8587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:10.603921 8786 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [term 2 FOLLOWER]: Refusing update from remote peer 317927315e3044b9bb73241a4e5b0d4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 07:02:10.603956 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.603781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46134)
0926 07:02:10.603846 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.603879 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 07:02:10.604216 9306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 07:02:10.604396 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.604257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.0.0.1:46150)
0926 07:02:10.604315 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.604369 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 07:02:10.604511 8572 consensus_peers.cc:487] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a -> Peer b9d6220d324442efb2c4200fa924ac73 (127.1.217.126:38595): Couldn't send request to peer b9d6220d324442efb2c4200fa924ac73. Status: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 07:02:10.604847 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317927315e3044b9bb73241a4e5b0d4a. Latest consensus state: current_term: 2 leader_uuid: "317927315e3044b9bb73241a4e5b0d4a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:10.604959 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:02:10.605423 9306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "317927315e3044b9bb73241a4e5b0d4a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:10.605525 9306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:10.605649 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "317927315e3044b9bb73241a4e5b0d4a" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "b9d6220d324442efb2c4200fa924ac73" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 38595 } } peers { permanent_uuid: "317927315e3044b9bb73241a4e5b0d4a" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34917 } attrs { promote: false } } peers { permanent_uuid: "21339434331f4ebcac499ec595ee3935" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 38647 } attrs { promote: false } } }
I20260926 07:02:10.605744 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21339434331f4ebcac499ec595ee3935 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:02:10.610682 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.1.217.67:37337)
0926 07:02:10.607776 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.607778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.607786 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:10.607811 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.607812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.607823 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.608038 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:10.608070 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608539 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.609853 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610353 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610446 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.610460 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.610572 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.610610 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":50,"threads_started":1}
I20260926 07:02:10.610682 9298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37337 (local address 127.0.0.1:37888)
0926 07:02:10.607393 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.607405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.607446 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.607911 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:10.607912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:10.607913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:10.607956 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.607957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608622 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.608623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.609756 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.609758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610446 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.610503 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.610506 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:10.610560 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 07:02:10.610601 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 07:02:10.610982 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43059 (local address 127.0.0.1:40520)
0926 07:02:10.607409 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.607420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.607466 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.607809 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:10.607810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:10.607811 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:10.607973 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.607974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608946 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.608947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.610030 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.610032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610779 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.610856 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.610860 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:10.610889 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 07:02:10.610926 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":128}
I20260926 07:02:10.611008 9162 tablet_service.cc:1467] Processing CreateTablet for tablet 5166813c8aca49578ed5db37e66b83d5 (DEFAULT_TABLE table=test-table [id=42c26a70de1a42888219b6228d94efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:10.611017 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.1.217.65:43787)
0926 07:02:10.607663 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.607665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.607671 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:02:10.607694 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.607695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.607706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.608071 (+ 365us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:10.608116 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608864 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.609834 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610382 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610869 (+ 487us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.610886 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.610905 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.610946 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":58,"threads_started":1}
I20260926 07:02:10.611152 9162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5166813c8aca49578ed5db37e66b83d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:10.611284 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.1.217.66:43059)
0926 07:02:10.607637 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.607640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.607649 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:10.607699 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.607700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.607711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.607837 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:10.608042 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608865 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610110 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610688 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.611112 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.611128 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.611150 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.611205 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260926 07:02:10.611531 9026 tablet_service.cc:1467] Processing CreateTablet for tablet 5166813c8aca49578ed5db37e66b83d5 (DEFAULT_TABLE table=test-table [id=42c26a70de1a42888219b6228d94efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:10.611658 9026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5166813c8aca49578ed5db37e66b83d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:10.612432 9318 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Bootstrap starting.
I20260926 07:02:10.612673 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.607394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43787 (local address 127.0.0.1:47980)
0926 07:02:10.607548 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.607558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.607593 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.607802 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:10.607803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:10.607806 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:10.607897 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.607900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.608972 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.608973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.609739 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:10.609742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.610478 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.610479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:10.610539 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.610544 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:02:10.610573 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 07:02:10.610613 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260926 07:02:10.612836 9318 tablet_bootstrap.cc:654] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:10.612960 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 5166813c8aca49578ed5db37e66b83d5 (DEFAULT_TABLE table=test-table [id=42c26a70de1a42888219b6228d94efc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:10.613071 8889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5166813c8aca49578ed5db37e66b83d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:10.613426 9318 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: No bootstrap required, opened a new log
I20260926 07:02:10.613461 9318 ts_tablet_manager.cc:1397] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:10.613570 9318 raft_consensus.cc:348] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.613615 9318 raft_consensus.cc:374] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:10.613627 9318 raft_consensus.cc:729] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daf3b140bf7349e8bd8cd78ce8f933fc, State: Initialized, Role: FOLLOWER
I20260926 07:02:10.613662 9318 consensus_queue.cc:260] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.613776 9318 ts_tablet_manager.cc:1428] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:10.614295 9322 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: Bootstrap starting.
I20260926 07:02:10.614741 9322 tablet_bootstrap.cc:654] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:10.615151 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.614425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:52673)
0926 07:02:10.615075 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.615115 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":78,"threads_started":1}
I20260926 07:02:10.615283 9320 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: Bootstrap starting.
I20260926 07:02:10.615293 9322 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: No bootstrap required, opened a new log
I20260926 07:02:10.615343 9322 ts_tablet_manager.cc:1397] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:10.615453 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.615371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:53121)
0926 07:02:10.615405 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.615429 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 07:02:10.615453 9322 raft_consensus.cc:348] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.615550 9322 raft_consensus.cc:374] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:10.615576 9322 raft_consensus.cc:729] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c25104d4b141ccbe833b90edb35618, State: Initialized, Role: FOLLOWER
I20260926 07:02:10.615628 9320 tablet_bootstrap.cc:654] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:10.615626 9322 consensus_queue.cc:260] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.615772 9322 ts_tablet_manager.cc:1428] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:10.616196 9320 tablet_bootstrap.cc:492] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: No bootstrap required, opened a new log
I20260926 07:02:10.616256 9320 ts_tablet_manager.cc:1397] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:10.616446 9320 raft_consensus.cc:348] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.616519 9320 raft_consensus.cc:374] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:10.616552 9320 raft_consensus.cc:729] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b11c6a9dffc4b04a7e929cc980b2303, State: Initialized, Role: FOLLOWER
I20260926 07:02:10.616607 9320 consensus_queue.cc:260] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.616775 9320 ts_tablet_manager.cc:1428] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:10.617029 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.615989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:45465)
0926 07:02:10.616961 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.616998 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":948,"thread_start_us":74,"threads_started":1}
I20260926 07:02:10.617318 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.616930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:60305)
0926 07:02:10.617231 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.617270 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":98,"threads_started":1}
I20260926 07:02:10.617355 9094 heartbeater.cc:503] Master 127.1.217.125:34917 was elected leader, sending a full tablet report...
I20260926 07:02:10.617408 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.617240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:34235)
0926 07:02:10.617357 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.617386 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 07:02:10.617745 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.617540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:50961)
0926 07:02:10.617676 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.617708 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":98,"threads_started":1}
I20260926 07:02:10.818002 9325 raft_consensus.cc:482] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:10.818138 9325 raft_consensus.cc:504] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.818367 9325 leader_election.cc:288] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337), 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:10.818624 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5166813c8aca49578ed5db37e66b83d5" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" is_pre_election: true
I20260926 07:02:10.818732 9182 raft_consensus.cc:2454] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 0.
I20260926 07:02:10.818653 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5166813c8aca49578ed5db37e66b83d5" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" is_pre_election: true
I20260926 07:02:10.818780 9046 raft_consensus.cc:2454] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 0.
I20260926 07:02:10.818933 8842 leader_election.cc:302] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c25104d4b141ccbe833b90edb35618, daf3b140bf7349e8bd8cd78ce8f933fc; no voters:
I20260926 07:02:10.819068 9325 raft_consensus.cc:2781] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:10.819108 9325 raft_consensus.cc:482] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:10.819123 9325 raft_consensus.cc:3037] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:10.819802 9325 raft_consensus.cc:504] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.819938 9325 leader_election.cc:288] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 election: Requested vote from peers daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337), 4b11c6a9dffc4b04a7e929cc980b2303 (127.1.217.66:43059)
I20260926 07:02:10.820089 9046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5166813c8aca49578ed5db37e66b83d5" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b11c6a9dffc4b04a7e929cc980b2303"
I20260926 07:02:10.820089 9182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5166813c8aca49578ed5db37e66b83d5" candidate_uuid: "62c25104d4b141ccbe833b90edb35618" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc"
I20260926 07:02:10.820176 9046 raft_consensus.cc:3037] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:10.820184 9182 raft_consensus.cc:3037] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:10.820983 9182 raft_consensus.cc:2454] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 1.
I20260926 07:02:10.820983 9046 raft_consensus.cc:2454] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62c25104d4b141ccbe833b90edb35618 in term 1.
I20260926 07:02:10.821187 8842 leader_election.cc:302] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c25104d4b141ccbe833b90edb35618, daf3b140bf7349e8bd8cd78ce8f933fc; no voters:
I20260926 07:02:10.821339 9325 raft_consensus.cc:2781] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:10.821545 9325 raft_consensus.cc:686] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 1 LEADER]: Becoming Leader. State: Replica: 62c25104d4b141ccbe833b90edb35618, State: Running, Role: LEADER
I20260926 07:02:10.821640 8957 heartbeater.cc:503] Master 127.1.217.125:34917 was elected leader, sending a full tablet report...
I20260926 07:02:10.821661 9325 consensus_queue.cc:237] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } }
I20260926 07:02:10.821904 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.821789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:47395)
0926 07:02:10.821826 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.821866 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 07:02:10.822280 8587 catalog_manager.cc:5495] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c25104d4b141ccbe833b90edb35618 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "62c25104d4b141ccbe833b90edb35618" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62c25104d4b141ccbe833b90edb35618" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43787 } health_report { overall_health: HEALTHY } } }
I20260926 07:02:10.836674 9316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.834644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.1.217.65:43787)
0926 07:02:10.834707 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:10.834709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:10.834718 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:10.834769 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:10.834769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:10.834778 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:10.834825 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:10.834940 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.835431 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.836114 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:10.836305 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:10.836356 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:10.836494 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:10.836518 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:10.836554 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:10.836603 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:02:10.836620 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:10.837648 9046 raft_consensus.cc:1261] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Refusing update from remote peer 62c25104d4b141ccbe833b90edb35618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:10.837663 9182 raft_consensus.cc:1261] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Refusing update from remote peer 62c25104d4b141ccbe833b90edb35618: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:10.837872 9331 consensus_queue.cc:1035] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b11c6a9dffc4b04a7e929cc980b2303" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:10.837980 9325 consensus_queue.cc:1035] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daf3b140bf7349e8bd8cd78ce8f933fc" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:10.838395 9230 heartbeater.cc:503] Master 127.1.217.125:34917 was elected leader, sending a full tablet report...
I20260926 07:02:10.838891 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.838746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:36861)
0926 07:02:10.838819 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.838850 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:02:10.838891 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.838804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:50305)
0926 07:02:10.838845 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.838866 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 07:02:10.839900 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.839802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:33319)
0926 07:02:10.839837 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.839866 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 07:02:10.840271 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.840166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.66:36777)
0926 07:02:10.840209 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.840242 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 07:02:10.840976 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.840895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.65:51241)
0926 07:02:10.840924 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.840949 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 07:02:10.841480 9323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.841291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:38595 (local address 127.1.217.67:56513)
0926 07:02:10.841421 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.841452 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:38595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 07:02:10.841813 8588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:10.841897 8588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:10.843096 8588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 317927315e3044b9bb73241a4e5b0d4a: Sending DeleteTablet for 3 replicas of tablet 5166813c8aca49578ed5db37e66b83d5
I20260926 07:02:10.843474 8889 tablet_service.cc:1514] Processing DeleteTablet for tablet 5166813c8aca49578ed5db37e66b83d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:10 UTC) from {username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:47980
I20260926 07:02:10.843511 9026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5166813c8aca49578ed5db37e66b83d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:10 UTC) from {username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:40520
I20260926 07:02:10.843499 9162 tablet_service.cc:1514] Processing DeleteTablet for tablet 5166813c8aca49578ed5db37e66b83d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:10 UTC) from {username='oryx', principal='oryx/127.1.217.125@KRBTEST.COM'} at 127.0.0.1:37888
I20260926 07:02:10.843662 9343 tablet_replica.cc:330] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: stopping tablet replica
I20260926 07:02:10.843699 9345 tablet_replica.cc:330] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: stopping tablet replica
I20260926 07:02:10.843719 9343 raft_consensus.cc:2229] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:10.843748 9345 raft_consensus.cc:2229] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:10.843772 9343 raft_consensus.cc:2258] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:10.843796 9345 raft_consensus.cc:2258] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:10.843878 9344 tablet_replica.cc:330] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: stopping tablet replica
I20260926 07:02:10.843921 9344 raft_consensus.cc:2229] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:10.843971 9344 raft_consensus.cc:2258] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:10.844035 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8736
I20260926 07:02:10.844179 9345 ts_tablet_manager.cc:1905] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:10.844208 9343 ts_tablet_manager.cc:1905] T 5166813c8aca49578ed5db37e66b83d5 P 62c25104d4b141ccbe833b90edb35618: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:10.845897 9345 ts_tablet_manager.cc:1918] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:10.845947 9345 log.cc:1196] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/5166813c8aca49578ed5db37e66b83d5
I20260926 07:02:10.846292 9345 ts_tablet_manager.cc:1939] T 5166813c8aca49578ed5db37e66b83d5 P daf3b140bf7349e8bd8cd78ce8f933fc: Deleting consensus metadata
I20260926 07:02:10.846597 8572 catalog_manager.cc:4841] TS daf3b140bf7349e8bd8cd78ce8f933fc (127.1.217.67:37337): tablet 5166813c8aca49578ed5db37e66b83d5 (table test-table [id=42c26a70de1a42888219b6228d94efc2]) successfully deleted
I20260926 07:02:10.851624 9344 ts_tablet_manager.cc:1905] T 5166813c8aca49578ed5db37e66b83d5 P 4b11c6a9dffc4b04a7e929cc980b2303: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:02:10.851747 9314 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:10.851530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43787 (local address 127.0.0.1:48004)
0926 07:02:10.851586 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.851599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:10.851638 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:10.851694 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:43787: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:02:10.851897 8574 catalog_manager.cc:4584] TS 62c25104d4b141ccbe833b90edb35618 (127.1.217.65:43787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5166813c8aca49578ed5db37e66b83d5: Network error: Client connection negotiation failed: client connection to 127.1.217.65:43787: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 07:02:10.851948 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8963
I20260926 07:02:10.856609 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 9099
I20260926 07:02:10.856701 9314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:10.856572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43059 (local address 127.0.0.1:40532)
0926 07:02:10.856633 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:10.856667 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.66:43059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:02:10.861096 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8455
I20260926 07:02:10.865757 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 8642
2026-09-26T07:02:10Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:10 dist-test-slave-t1mc krb5kdc[9348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:10 dist-test-slave-t1mc krb5kdc[9348](info): set up 2 sockets
Sep 26 07:02:10 dist-test-slave-t1mc krb5kdc[9348](info): commencing operation
krb5kdc: starting...
W20260926 07:02:11.901301 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:11 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab.
Entry for principal oryx/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab.
I20260926 07:02:12.019467 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45407
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab
--principal=oryx/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:45407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:12.099426 9361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:12.099668 9361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:12.099705 9361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:12.100911 9361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:12.100977 9361 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 07:02:12.100999 9361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:12.101020 9361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:12.101051 9361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:12.101078 9361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:12.102589 9361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:45407
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.126.keytab
--principal=oryx/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:12.102871 9361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:12.103094 9361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:12.107880 9369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.107904 9367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.107880 9366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:12.107909 9361 server_base.cc:1034] running on GCE node
I20260926 07:02:12.108469 9361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:12.108755 9361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:12.109946 9361 hybrid_clock.cc:648] HybridClock initialized: now 1790406132109929 us; error 42 us; skew 500 ppm
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:12.112322 9361 init.cc:377] Logged in from keytab as oryx/127.1.217.126@KRBTEST.COM (short username oryx)
I20260926 07:02:12.112864 9361 webserver.cc:460] Webserver started at http://127.1.217.126:46121/ using document root <none> and password file <none>
I20260926 07:02:12.113090 9361 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:12.113155 9361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:12.113308 9361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:12.114359 9361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "ae50e1b7571741e1b158b68b17bd7f4d"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "c006eaa513b5c5a9433d5ac487665961"
server_key_iv: "23f66b4c0bfa292833c290fe5e3a56e4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.114746 9361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "ae50e1b7571741e1b158b68b17bd7f4d"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "c006eaa513b5c5a9433d5ac487665961"
server_key_iv: "23f66b4c0bfa292833c290fe5e3a56e4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.116055 9361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:02:12.116832 9376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.117022 9361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.117122 9361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "ae50e1b7571741e1b158b68b17bd7f4d"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "c006eaa513b5c5a9433d5ac487665961"
server_key_iv: "23f66b4c0bfa292833c290fe5e3a56e4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.117209 9361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:12.132128 9361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:12.134367 9361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:12.134541 9361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:12.139330 9361 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:45407
I20260926 07:02:12.139376 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:45407 every 8 connection(s)
I20260926 07:02:12.139727 9361 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:12.140383 9439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.142587 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Bootstrap starting.
I20260926 07:02:12.143133 9439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.143378 9439 log.cc:824] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:12.144105 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: No bootstrap required, opened a new log
I20260926 07:02:12.144114 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 9361
I20260926 07:02:12.144196 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:12.144424 1893 external_mini_cluster.cc:1428] Setting key ea2cc08f399fef83691770eead4c734b
I20260926 07:02:12.146556 9439 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } }
I20260926 07:02:12.146727 9439 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.146768 9439 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae50e1b7571741e1b158b68b17bd7f4d, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.146926 9439 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } }
I20260926 07:02:12.147019 9439 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:12.147053 9439 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:12.147107 9439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.147779 9439 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } }
I20260926 07:02:12.147866 9439 leader_election.cc:302] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae50e1b7571741e1b158b68b17bd7f4d; no voters:
I20260926 07:02:12.148031 9439 leader_election.cc:288] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:12.148075 9444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:12.148196 9444 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [term 1 LEADER]: Becoming Leader. State: Replica: ae50e1b7571741e1b158b68b17bd7f4d, State: Running, Role: LEADER
I20260926 07:02:12.148293 9439 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:12.148272 9444 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } }
I20260926 07:02:12.148727 9445 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } } }
I20260926 07:02:12.148890 9445 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:12.149123 9446 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae50e1b7571741e1b158b68b17bd7f4d. Latest consensus state: current_term: 1 leader_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae50e1b7571741e1b158b68b17bd7f4d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45407 } } }
I20260926 07:02:12.149492 9446 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:12.149648 9453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:12.150173 9453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:12.151722 9453 catalog_manager.cc:1347] Generated new cluster ID: 565b70c2a1b24eb18a783fb96afabc70
I20260926 07:02:12.151775 9453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.157231 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.145016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.1.217.126:45407)
0926 07:02:12.145181 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.145189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.145214 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:02:12.146535 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.146541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.146589 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.146707 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.147519 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.148846 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.149971 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.150562 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.150673 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.153500 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.153505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.153505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.153517 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.155570 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.155959 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.155962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.155963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.156087 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.156296 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.156297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.156297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.156610 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.156800 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.156883 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.157024 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 07:02:12.171098 9453 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:12.171674 9453 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:12.174037 9453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Generated new TSK 0
I20260926 07:02:12.174226 9453 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
I20260926 07:02:12.203591 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45407
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:12.290371 9465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:12.290593 9465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:12.290618 9465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:12.291778 9465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:12.291821 9465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:12.291841 9465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:02:12.291886 9465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:12.293397 9465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:12.293645 9465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:12.293884 9465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:12.298693 9470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.298689 9471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.298743 9473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:12.298853 9465 server_base.cc:1034] running on GCE node
I20260926 07:02:12.299067 9465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:12.299311 9465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:12.300455 9465 hybrid_clock.cc:648] HybridClock initialized: now 1790406132300436 us; error 35 us; skew 500 ppm
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:12.303027 9465 init.cc:377] Logged in from keytab as oryx/127.1.217.65@KRBTEST.COM (short username oryx)
I20260926 07:02:12.303507 9465 webserver.cc:460] Webserver started at http://127.1.217.65:44621/ using document root <none> and password file <none>
I20260926 07:02:12.303730 9465 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:12.303776 9465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:12.303946 9465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:12.305078 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "6861959438e74547a2e40553b3e2924a"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "a621dd79cbd890d0f1153f52446daec7"
server_key_iv: "e4f5a46911d5688eaaf4a31819ac29b4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.305454 9465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "6861959438e74547a2e40553b3e2924a"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "a621dd79cbd890d0f1153f52446daec7"
server_key_iv: "e4f5a46911d5688eaaf4a31819ac29b4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.306797 9465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:12.307492 9480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.307663 9465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:12.307770 9465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "6861959438e74547a2e40553b3e2924a"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "a621dd79cbd890d0f1153f52446daec7"
server_key_iv: "e4f5a46911d5688eaaf4a31819ac29b4"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.307848 9465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:12.331874 9465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:12.334332 9465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:12.334503 9465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:12.335119 9465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:12.335168 9465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.335214 9465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:12.335235 9465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.342730 9465 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41239
I20260926 07:02:12.343075 9465 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:12.343478 9595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41239 every 8 connection(s)
I20260926 07:02:12.345640 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.335521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:37101 (local address 127.1.217.126:45407)
0926 07:02:12.335583 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.335586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.335846 (+ 260us) server_negotiation.cc:407] Connection header received
0926 07:02:12.336614 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.336616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.336625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.336670 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.337623 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.338119 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.339195 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.339418 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.339496 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.342756 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.342758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.342758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.342768 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.344237 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.344669 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.344671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.344671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.344734 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.345000 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.345001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.345001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.345119 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.345195 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.345464 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.345547 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:02:12.345841 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.335511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.65:37101)
0926 07:02:12.335700 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.335766 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.336477 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.336784 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.336789 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.337203 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.337519 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.337526 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.338217 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.338218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.339113 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.339115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.339503 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.339504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.339633 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.340188 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.340198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.342660 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.344345 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.344352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.344354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.344584 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.344817 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.344817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.344818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.344929 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.345186 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.345191 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.345333 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.345443 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.345597 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260926 07:02:12.346318 9596 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45407
I20260926 07:02:12.346401 9596 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:12.346589 9596 heartbeater.cc:511] Master 127.1.217.126:45407 requested a full tablet report, sending...
I20260926 07:02:12.347151 9392 ts_manager.cc:194] Registered new tserver with Master: 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239)
I20260926 07:02:12.348100 9392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:37101
I20260926 07:02:12.349251 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 9465
I20260926 07:02:12.349401 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:12.349640 1893 external_mini_cluster.cc:1428] Setting key 8c0bf753e1f2bafadb3f15786e4784ed
I20260926 07:02:12.354001 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.347344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:50405 (local address 127.1.217.126:45407)
0926 07:02:12.347383 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.347385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.347445 (+ 60us) server_negotiation.cc:407] Connection header received
0926 07:02:12.347515 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.347516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.347523 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.347597 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.347926 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.348568 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.349602 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.349905 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.349969 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.351245 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.351247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.351247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.351254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.352886 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.353222 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.353223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.353224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.353275 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.353479 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.353480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.353481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.353589 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.353664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.353829 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.353906 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:12.354015 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.347261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.65:50405)
0926 07:02:12.347381 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.347393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.347454 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.347577 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.347608 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.347801 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.347867 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.347868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.348662 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.348663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.349521 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.349523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.350235 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.350236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.350305 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.350732 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.350738 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.351154 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.352971 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.352972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.352972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.353138 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.353350 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.353350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.353351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.353409 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.353656 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.353657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.353699 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.353805 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.353907 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
I20260926 07:02:12.374949 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45407
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:12.455788 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:12.455989 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:12.456014 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:12.457211 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:12.457252 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:12.457273 9605 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:02:12.457329 9605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:12.458810 9605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:12.459020 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:12.459228 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:02:12.464722 9605 server_base.cc:1034] running on GCE node
W20260926 07:02:12.464686 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.464689 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.464669 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:12.465162 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:12.465408 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:12.466574 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790406132466559 us; error 29 us; skew 500 ppm
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:12.469013 9605 init.cc:377] Logged in from keytab as oryx/127.1.217.66@KRBTEST.COM (short username oryx)
I20260926 07:02:12.469458 9605 webserver.cc:460] Webserver started at http://127.1.217.66:45065/ using document root <none> and password file <none>
I20260926 07:02:12.469677 9605 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:12.469745 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:12.469895 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:12.470912 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "b40e342bda0d40b8b086a710bbb6e170"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "73487cc1903971685a05670b4a94d05f"
server_key_iv: "aa217f6c488d516ab7743b1c31f2fd2c"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.471271 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "b40e342bda0d40b8b086a710bbb6e170"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "73487cc1903971685a05670b4a94d05f"
server_key_iv: "aa217f6c488d516ab7743b1c31f2fd2c"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.472527 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 07:02:12.473309 9620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.473485 9605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.473582 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "b40e342bda0d40b8b086a710bbb6e170"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "73487cc1903971685a05670b4a94d05f"
server_key_iv: "aa217f6c488d516ab7743b1c31f2fd2c"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.473667 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:12.498659 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:12.501135 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:12.501312 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:12.501839 9605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:12.501899 9605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.501940 9605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:12.501973 9605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.509135 9605 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:41309
I20260926 07:02:12.509231 9735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:41309 every 8 connection(s)
I20260926 07:02:12.509582 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:12.510087 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 9605
I20260926 07:02:12.510205 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:12.510439 1893 external_mini_cluster.cc:1428] Setting key 596256ebba135b42702f4d2160befa75
I20260926 07:02:12.513496 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.502474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50729 (local address 127.1.217.126:45407)
0926 07:02:12.502544 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.502547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.502750 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:02:12.503349 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.503351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.503363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.503453 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.504391 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.505297 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.506326 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.506679 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.506757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.509522 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.509523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.509523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.509531 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.511938 (+ 2407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.512475 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.512477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.512477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.512546 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.512833 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.512834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.512834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.512951 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.513030 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.513319 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.513399 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:12.513695 9629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.502443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.66:50729)
0926 07:02:12.502611 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.502651 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.503199 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.503535 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.503539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.503914 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.504262 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.504274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.505422 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.505424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.506242 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.506244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.506785 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.506786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.506888 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.507867 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.507875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.509420 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.512025 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.512032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.512035 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.512376 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.512616 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.512617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.512617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.512723 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.513023 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.513029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.513177 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.513297 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.513464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260926 07:02:12.514199 9736 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45407
I20260926 07:02:12.514281 9736 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:12.514420 9736 heartbeater.cc:511] Master 127.1.217.126:45407 requested a full tablet report, sending...
I20260926 07:02:12.514766 9393 ts_manager.cc:194] Registered new tserver with Master: b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309)
I20260926 07:02:12.515157 9393 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:50729
WARNING: no policy specified for oryx/127.1.217.67@KRBTEST.COM; defaulting to no policy
I20260926 07:02:12.521464 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.515509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:44261 (local address 127.1.217.126:45407)
0926 07:02:12.515548 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.515550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.515605 (+ 55us) server_negotiation.cc:407] Connection header received
0926 07:02:12.515693 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.515693 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.515703 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.515767 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.516050 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.516883 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.517591 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.517899 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.517972 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.518924 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.518925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.518926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.518933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.520396 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.520739 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.520740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.520740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.520849 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.521053 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.521053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.521054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.521186 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.521240 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.521352 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.521406 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:12.521476 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.515361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.66:44261)
0926 07:02:12.515538 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.515551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.515637 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.515769 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.515770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.515920 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.515979 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.515980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.516972 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.516973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.517508 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.517509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.517976 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.517977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.518055 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.518537 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.518541 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.518835 (+ 294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.520484 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.520485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.520485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.520644 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.520917 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.520918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.520918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.520992 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.521240 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.521241 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.521277 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.521328 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.521405 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
I20260926 07:02:12.535948 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45407
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:36745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:12.616077 9745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:12.616318 9745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:12.616358 9745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:12.617621 9745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:12.617691 9745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:12.617728 9745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 07:02:12.617785 9745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:12.619267 9745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.1.217.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:12.619534 9745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:12.619807 9745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:12.624735 9750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.624728 9751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:12.624807 9753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:12.625072 9745 server_base.cc:1034] running on GCE node
I20260926 07:02:12.625301 9745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:12.625540 9745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:12.626708 9745 hybrid_clock.cc:648] HybridClock initialized: now 1790406132626695 us; error 23 us; skew 500 ppm
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:12.629308 9745 init.cc:377] Logged in from keytab as oryx/127.1.217.67@KRBTEST.COM (short username oryx)
I20260926 07:02:12.629774 9745 webserver.cc:460] Webserver started at http://127.1.217.67:38991/ using document root <none> and password file <none>
I20260926 07:02:12.630005 9745 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:12.630081 9745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:12.630234 9745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:12.631270 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "1abe7435fac44c9d86ad86dfa15b2060"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "bba90d333f570b7f98427b0efe073043"
server_key_iv: "0a743b3308fdc50c5f64d12d7872baa7"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.631630 9745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "1abe7435fac44c9d86ad86dfa15b2060"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "bba90d333f570b7f98427b0efe073043"
server_key_iv: "0a743b3308fdc50c5f64d12d7872baa7"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.632946 9745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:12.633699 9760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.633874 9745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:12.633980 9745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "1abe7435fac44c9d86ad86dfa15b2060"
format_stamp: "Formatted at 2026-09-26 07:02:12 on dist-test-slave-t1mc"
server_key: "bba90d333f570b7f98427b0efe073043"
server_key_iv: "0a743b3308fdc50c5f64d12d7872baa7"
server_key_version: "encryptionkey@0"
I20260926 07:02:12.634085 9745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:12.641675 9745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:12.643914 9745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:12.644093 9745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:12.644613 9745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:12.644670 9745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.644716 9745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:12.644738 9745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.652345 9745 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:34629
I20260926 07:02:12.652426 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:34629 every 8 connection(s)
I20260926 07:02:12.652745 9745 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:12.655068 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.645016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:44123 (local address 127.1.217.126:45407)
0926 07:02:12.645064 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.645067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.645324 (+ 257us) server_negotiation.cc:407] Connection header received
0926 07:02:12.646042 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.646043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.646054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.646118 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.647065 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.647817 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.649174 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.649459 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.649527 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.652140 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.652141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.652142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.652150 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.653647 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.653988 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.653988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.653989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.654036 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.654268 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.654268 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.654269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.654417 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.654492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.654889 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.654985 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:02:12.655272 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.645019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.67:44123)
0926 07:02:12.645168 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.645218 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.645934 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.646179 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.646184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.646646 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.646960 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.646968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.647917 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.647920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.649069 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.649072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.649577 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.649578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.649716 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.650252 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.650262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.652041 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.653719 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.653723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.653725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.653924 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.654089 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.654090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.654090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.654212 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.654512 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.654517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.654713 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.654840 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.655040 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260926 07:02:12.655723 9876 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45407
I20260926 07:02:12.655803 9876 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:12.655999 9876 heartbeater.cc:511] Master 127.1.217.126:45407 requested a full tablet report, sending...
I20260926 07:02:12.656352 9392 ts_manager.cc:194] Registered new tserver with Master: 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:12.656854 9392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:44123
I20260926 07:02:12.660598 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 9745
I20260926 07:02:12.660697 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:12.660904 1893 external_mini_cluster.cc:1428] Setting key 91832719157d2155b2685124d42d1a69
I20260926 07:02:12.661824 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:02:12.662156 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.656678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.1.217.67:33933)
0926 07:02:12.656807 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.656818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.656875 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.657035 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.657036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.657236 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.657310 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.657311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.657968 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.657969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.658451 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.658452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.658753 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.658754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.658800 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.659137 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.659141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.659392 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.661141 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.661142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.661143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.661361 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.661559 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.661560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.661560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.661626 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.661902 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.661903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.661951 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.662019 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.662095 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260926 07:02:12.662202 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.656735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:33933 (local address 127.1.217.126:45407)
0926 07:02:12.656801 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.656802 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.656858 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:02:12.656945 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.656945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.656955 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.657025 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.657390 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.657878 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.658517 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.658693 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.658740 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.659462 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.659463 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.659463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.659468 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.661056 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.661432 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.661433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.661433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.661489 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.661706 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.661707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.661707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.661829 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.661893 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.662060 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.662126 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:02:12.682888 9443 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:12.677739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.1.217.126:45407)
0926 07:02:12.677798 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.677800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.677809 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:12.677874 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.677875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.677884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.677936 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.678463 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.678937 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.679634 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.679808 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.679855 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.682789 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.689867 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.683597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.1.217.126:45407)
0926 07:02:12.683639 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.683640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.683728 (+ 88us) server_negotiation.cc:407] Connection header received
0926 07:02:12.683765 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.683766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.683773 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.683824 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.684283 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.684748 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.685475 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.685651 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.685707 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.687685 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.687687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.687687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.687694 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.688922 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.689217 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.689218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.689218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.689268 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.689470 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.689471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.689471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.689582 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.689661 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.689746 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.689802 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:02:12.691648 9392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:12.693259 9392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:12.699487 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:41309 (local address 127.0.0.1:33294)
0926 07:02:12.695630 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.695641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.695697 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.696042 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.696045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.696048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.696115 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.696117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.697073 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.697074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.698249 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.698251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699162 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.699240 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699262 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.699311 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.699360 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260926 07:02:12.699512 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.1.217.67:34629)
0926 07:02:12.695931 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.695935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.695945 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:12.695985 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.695987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.696002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.696113 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.696293 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.697389 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.698554 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699049 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699135 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699213 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.699374 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.699412 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260926 07:02:12.699643 9899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41239 (local address 127.0.0.1:46894)
0926 07:02:12.695575 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.695618 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.695697 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.696311 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.696311 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.696314 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.696358 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.696360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.697579 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.697580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.698570 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.698572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699451 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.699510 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699513 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.699546 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.699585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":56,"threads_started":1}
I20260926 07:02:12.699708 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.1.217.66:41309)
0926 07:02:12.695861 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.695866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.695877 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:02:12.695916 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.695918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.695939 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.696037 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.696204 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.696988 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.698338 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699064 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699448 (+ 384us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699546 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.699571 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.699609 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260926 07:02:12.700033 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46894 (local address 127.1.217.65:41239)
0926 07:02:12.695950 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.695955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.695964 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:12.696171 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.696174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.696189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.696431 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.696486 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.697504 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.698659 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699361 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699831 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699904 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.699923 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.699950 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":76,"threads_started":1}
I20260926 07:02:12.700165 9900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.695430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:34629 (local address 127.0.0.1:50656)
0926 07:02:12.695797 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.695804 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.695841 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.696112 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.696113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.696114 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.696156 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.696158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.697488 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.697489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.698472 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.698474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.699145 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.699146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.699244 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.699264 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.699355 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.699398 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":94,"threads_started":1}
I20260926 07:02:12.700547 9530 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ae265543146f9b3a49a031a085a73 (DEFAULT_TABLE table=test-table [id=f2f4313796ed477aba76c7a28413ad7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.700551 9670 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ae265543146f9b3a49a031a085a73 (DEFAULT_TABLE table=test-table [id=f2f4313796ed477aba76c7a28413ad7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.700827 9810 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ae265543146f9b3a49a031a085a73 (DEFAULT_TABLE table=test-table [id=f2f4313796ed477aba76c7a28413ad7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.700958 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ae265543146f9b3a49a031a085a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.701133 9670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ae265543146f9b3a49a031a085a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.701174 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ae265543146f9b3a49a031a085a73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.703831 9904 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Bootstrap starting.
I20260926 07:02:12.704478 9904 tablet_bootstrap.cc:654] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.704905 9905 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Bootstrap starting.
I20260926 07:02:12.704977 9904 log.cc:824] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:12.705509 9905 tablet_bootstrap.cc:654] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.705698 9904 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: No bootstrap required, opened a new log
I20260926 07:02:12.705770 9904 ts_tablet_manager.cc:1397] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:02:12.705835 9905 log.cc:824] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:12.704753 9906 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Bootstrap starting.
I20260926 07:02:12.706477 9905 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: No bootstrap required, opened a new log
I20260926 07:02:12.706557 9905 ts_tablet_manager.cc:1397] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:12.706651 9906 tablet_bootstrap.cc:654] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.706969 9906 log.cc:824] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:12.707978 9906 tablet_bootstrap.cc:492] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: No bootstrap required, opened a new log
I20260926 07:02:12.708045 9906 ts_tablet_manager.cc:1397] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:02:12.707933 9904 raft_consensus.cc:348] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.708097 9904 raft_consensus.cc:374] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.708144 9904 raft_consensus.cc:729] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.708277 9904 consensus_queue.cc:260] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.708413 9905 raft_consensus.cc:348] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.708539 9904 ts_tablet_manager.cc:1428] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:02:12.708575 9905 raft_consensus.cc:374] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.708626 9905 raft_consensus.cc:729] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e342bda0d40b8b086a710bbb6e170, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.708648 9876 heartbeater.cc:503] Master 127.1.217.126:45407 was elected leader, sending a full tablet report...
I20260926 07:02:12.708743 9905 consensus_queue.cc:260] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.709016 9905 ts_tablet_manager.cc:1428] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:02:12.709128 9736 heartbeater.cc:503] Master 127.1.217.126:45407 was elected leader, sending a full tablet report...
I20260926 07:02:12.710088 9906 raft_consensus.cc:348] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.710201 9906 raft_consensus.cc:374] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.710223 9906 raft_consensus.cc:729] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.710350 9906 consensus_queue.cc:260] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.710595 9906 ts_tablet_manager.cc:1428] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:02:12.710660 9596 heartbeater.cc:503] Master 127.1.217.126:45407 was elected leader, sending a full tablet report...
W20260926 07:02:12.760193 9737 tablet.cc:2369] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:12.772845 9910 raft_consensus.cc:482] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:12.772958 9910 raft_consensus.cc:504] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.773289 9910 leader_election.cc:288] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239), b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309)
I20260926 07:02:12.776022 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.773443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41239 (local address 127.1.217.67:55807)
0926 07:02:12.773520 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.773533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.773575 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.773693 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.773695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.773696 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.773769 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.773771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.774416 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.774417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.775161 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.775164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.775724 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.775820 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.775825 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.775884 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.775932 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 07:02:12.776046 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.773529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:55807 (local address 127.1.217.65:41239)
0926 07:02:12.773583 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.773585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.773596 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:02:12.773626 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.773627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.773637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.773691 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.773850 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.774326 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775263 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.775637 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775724 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.775733 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.775908 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.775957 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 07:02:12.776273 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.773609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60535 (local address 127.1.217.66:41309)
0926 07:02:12.773665 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.773667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.773766 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:02:12.773844 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.773845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.773853 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.773926 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.774081 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.774533 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775439 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.775912 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.775990 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.776159 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.776215 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:02:12.776378 9550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ae265543146f9b3a49a031a085a73" candidate_uuid: "1abe7435fac44c9d86ad86dfa15b2060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6861959438e74547a2e40553b3e2924a" is_pre_election: true
I20260926 07:02:12.776532 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.773482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:41309 (local address 127.1.217.67:60535)
0926 07:02:12.773683 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.773693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.773783 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.773928 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.773929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.773931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.774016 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.774018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.774623 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.774624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.775351 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.775353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.775992 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.775993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.776061 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.776065 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.776407 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.776445 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":128,"threads_started":1}
I20260926 07:02:12.776580 9550 raft_consensus.cc:2454] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abe7435fac44c9d86ad86dfa15b2060 in term 0.
I20260926 07:02:12.776823 9761 leader_election.cc:302] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abe7435fac44c9d86ad86dfa15b2060, 6861959438e74547a2e40553b3e2924a; no voters:
I20260926 07:02:12.776872 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ae265543146f9b3a49a031a085a73" candidate_uuid: "1abe7435fac44c9d86ad86dfa15b2060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170" is_pre_election: true
I20260926 07:02:12.776978 9910 raft_consensus.cc:2781] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:12.777029 9910 raft_consensus.cc:482] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:12.777035 9690 raft_consensus.cc:2454] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1abe7435fac44c9d86ad86dfa15b2060 in term 0.
I20260926 07:02:12.777061 9910 raft_consensus.cc:3037] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.777956 9910 raft_consensus.cc:504] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.778108 9910 leader_election.cc:288] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [CANDIDATE]: Term 1 election: Requested vote from peers 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239), b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309)
I20260926 07:02:12.778272 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ae265543146f9b3a49a031a085a73" candidate_uuid: "1abe7435fac44c9d86ad86dfa15b2060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170"
I20260926 07:02:12.778272 9550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ae265543146f9b3a49a031a085a73" candidate_uuid: "1abe7435fac44c9d86ad86dfa15b2060" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6861959438e74547a2e40553b3e2924a"
I20260926 07:02:12.778352 9550 raft_consensus.cc:3037] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.778352 9690 raft_consensus.cc:3037] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.779214 9690 raft_consensus.cc:2454] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abe7435fac44c9d86ad86dfa15b2060 in term 1.
I20260926 07:02:12.779219 9550 raft_consensus.cc:2454] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1abe7435fac44c9d86ad86dfa15b2060 in term 1.
I20260926 07:02:12.779366 9761 leader_election.cc:302] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abe7435fac44c9d86ad86dfa15b2060, 6861959438e74547a2e40553b3e2924a; no voters:
I20260926 07:02:12.779465 9910 raft_consensus.cc:2781] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:12.779659 9910 raft_consensus.cc:686] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 LEADER]: Becoming Leader. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Running, Role: LEADER
I20260926 07:02:12.779752 9910 consensus_queue.cc:237] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.780546 9393 catalog_manager.cc:5495] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67). New cstate: current_term: 1 leader_uuid: "1abe7435fac44c9d86ad86dfa15b2060" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } health_report { overall_health: HEALTHY } } }
I20260926 07:02:12.795236 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.792995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50666 (local address 127.1.217.67:34629)
0926 07:02:12.793056 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.793057 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.793069 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:02:12.793146 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.793146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.793162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.793224 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:12.793340 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.793791 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.794525 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.794698 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.794753 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.794929 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:12.795020 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:12.795078 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.795167 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.795183 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:02:12.796944 9550 raft_consensus.cc:1261] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Refusing update from remote peer 1abe7435fac44c9d86ad86dfa15b2060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:12.796970 9690 raft_consensus.cc:1261] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Refusing update from remote peer 1abe7435fac44c9d86ad86dfa15b2060: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:12.797310 9914 consensus_queue.cc:1035] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:12.797463 9910 consensus_queue.cc:1035] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:12.800172 9917 mvcc.cc:205] Tried to move back new op lower bound from 7333503519933812736 to 7333503519866572800. Current Snapshot: MvccSnapshot[applied={T|T < 7333503519933812736 or (T in {7333503519933812736})}]
I20260926 07:02:12.801579 9921 mvcc.cc:205] Tried to move back new op lower bound from 7333503519933812736 to 7333503519866572800. Current Snapshot: MvccSnapshot[applied={T|T < 7333503519933812736}]
I20260926 07:02:12.801656 9392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:12.801743 9392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:12.802565 9392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Sending DeleteTablet for 3 replicas of tablet 1c8ae265543146f9b3a49a031a085a73
I20260926 07:02:12.802922 9670 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ae265543146f9b3a49a031a085a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:12 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33294
I20260926 07:02:12.802937 9810 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ae265543146f9b3a49a031a085a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:12 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:50656
I20260926 07:02:12.803082 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ae265543146f9b3a49a031a085a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:12 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33294
I20260926 07:02:12.803187 9530 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ae265543146f9b3a49a031a085a73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:12 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:46894
I20260926 07:02:12.803215 9928 tablet_replica.cc:330] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: stopping tablet replica
I20260926 07:02:12.803310 9928 raft_consensus.cc:2229] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:12.803411 9928 raft_consensus.cc:2258] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:12.803434 9929 tablet_replica.cc:330] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: stopping tablet replica
I20260926 07:02:12.803604 9929 raft_consensus.cc:2229] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:12.803594 9392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:12.803737 9392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:12.803889 9928 ts_tablet_manager.cc:1905] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:12.804003 9929 raft_consensus.cc:2258] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:12.804473 9929 ts_tablet_manager.cc:1905] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:12.805035 9810 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2c1619d0644aaab948de9b94125a97 (DEFAULT_TABLE table=test-table [id=6bad7d2e323d4eefb24d691480d7210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.805154 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2c1619d0644aaab948de9b94125a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.805733 9928 ts_tablet_manager.cc:1918] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:12.805814 9928 log.cc:1196] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/1c8ae265543146f9b3a49a031a085a73
I20260926 07:02:12.806066 9928 ts_tablet_manager.cc:1939] T 1c8ae265543146f9b3a49a031a085a73 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting consensus metadata
I20260926 07:02:12.806357 9904 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Bootstrap starting.
I20260926 07:02:12.806761 9904 tablet_bootstrap.cc:654] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.806794 9929 ts_tablet_manager.cc:1918] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:12.806888 9929 log.cc:1196] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/1c8ae265543146f9b3a49a031a085a73
I20260926 07:02:12.807173 9379 catalog_manager.cc:4841] TS 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629): tablet 1c8ae265543146f9b3a49a031a085a73 (table test-table [id=f2f4313796ed477aba76c7a28413ad7a]) successfully deleted
I20260926 07:02:12.807196 9929 ts_tablet_manager.cc:1939] T 1c8ae265543146f9b3a49a031a085a73 P 6861959438e74547a2e40553b3e2924a: Deleting consensus metadata
I20260926 07:02:12.807417 9669 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2c1619d0644aaab948de9b94125a97 (DEFAULT_TABLE table=test-table [id=6bad7d2e323d4eefb24d691480d7210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.807441 9904 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: No bootstrap required, opened a new log
I20260926 07:02:12.807482 9904 ts_tablet_manager.cc:1397] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:12.807526 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2c1619d0644aaab948de9b94125a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.807611 9904 raft_consensus.cc:348] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.807657 9904 raft_consensus.cc:374] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.807670 9904 raft_consensus.cc:729] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.807736 9904 consensus_queue.cc:260] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.807835 9904 ts_tablet_manager.cc:1428] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.808934 9905 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Bootstrap starting.
I20260926 07:02:12.809022 9377 catalog_manager.cc:4841] TS 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239): tablet 1c8ae265543146f9b3a49a031a085a73 (table test-table [id=f2f4313796ed477aba76c7a28413ad7a]) successfully deleted
I20260926 07:02:12.809309 9927 tablet_replica.cc:330] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: stopping tablet replica
I20260926 07:02:12.809306 9530 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2c1619d0644aaab948de9b94125a97 (DEFAULT_TABLE table=test-table [id=6bad7d2e323d4eefb24d691480d7210d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:12.809374 9905 tablet_bootstrap.cc:654] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.809402 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2c1619d0644aaab948de9b94125a97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.809526 9927 raft_consensus.cc:2229] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:12.809675 9927 raft_consensus.cc:2258] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:12.809973 9905 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: No bootstrap required, opened a new log
I20260926 07:02:12.810020 9905 ts_tablet_manager.cc:1397] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.810156 9905 raft_consensus.cc:348] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.810243 9905 raft_consensus.cc:374] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.810271 9905 raft_consensus.cc:729] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e342bda0d40b8b086a710bbb6e170, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.810326 9905 consensus_queue.cc:260] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.810412 9905 ts_tablet_manager.cc:1428] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:12.810523 9906 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Bootstrap starting.
I20260926 07:02:12.810931 9906 tablet_bootstrap.cc:654] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.810958 9927 ts_tablet_manager.cc:1905] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:12.811566 9906 tablet_bootstrap.cc:492] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: No bootstrap required, opened a new log
I20260926 07:02:12.811630 9906 ts_tablet_manager.cc:1397] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.811780 9906 raft_consensus.cc:348] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.811864 9906 raft_consensus.cc:374] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.811892 9906 raft_consensus.cc:729] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.811933 9906 consensus_queue.cc:260] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.812031 9906 ts_tablet_manager.cc:1428] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.812546 9927 ts_tablet_manager.cc:1918] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:12.812628 9927 log.cc:1196] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/1c8ae265543146f9b3a49a031a085a73
I20260926 07:02:12.812906 9927 ts_tablet_manager.cc:1939] T 1c8ae265543146f9b3a49a031a085a73 P b40e342bda0d40b8b086a710bbb6e170: Deleting consensus metadata
I20260926 07:02:12.813285 9380 catalog_manager.cc:4841] TS b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309): tablet 1c8ae265543146f9b3a49a031a085a73 (table test-table [id=f2f4313796ed477aba76c7a28413ad7a]) successfully deleted
W20260926 07:02:12.813421 9380 catalog_manager.cc:4808] TS b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309): delete failed for tablet 1c8ae265543146f9b3a49a031a085a73 because the tablet was not found. No further retry: Not found: Tablet not found: 1c8ae265543146f9b3a49a031a085a73
W20260926 07:02:12.843883 9597 tablet.cc:2369] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:02:12.890285 9912 raft_consensus.cc:482] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:12.890399 9912 raft_consensus.cc:504] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.890780 9912 leader_election.cc:288] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:12.894663 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.890987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:41309 (local address 127.1.217.65:34889)
0926 07:02:12.891188 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.891197 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.891245 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.891389 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.891390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.891392 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.891481 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.891484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.892429 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.892431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.893653 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.893657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.894355 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.894356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.894440 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.894446 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.894507 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.894560 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260926 07:02:12.894662 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.891120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55377 (local address 127.1.217.67:34629)
0926 07:02:12.891164 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.891167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.891226 (+ 59us) server_negotiation.cc:407] Connection header received
0926 07:02:12.891339 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.891340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.891352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.891402 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.891581 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.892300 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.893447 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.894255 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.894335 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.894344 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.894566 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.894604 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:02:12.894659 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.891208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:34889 (local address 127.1.217.66:41309)
0926 07:02:12.891245 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.891247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.891256 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:12.891296 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.891297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.891306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.891387 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:12.891572 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.892317 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.893754 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.894255 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.894335 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.894344 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.894535 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.894586 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:12.895018 9933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.890987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:34629 (local address 127.1.217.65:55377)
0926 07:02:12.891133 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.891146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.891253 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.891439 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.891440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.891442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:12.891488 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.891490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.892402 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.892403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.893349 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.893353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.894406 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.894407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.894459 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.894463 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.894843 (+ 380us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.894903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":48,"spinlock_wait_cycles":896,"thread_start_us":78,"threads_started":1}
I20260926 07:02:12.895103 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2c1619d0644aaab948de9b94125a97" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170" is_pre_election: true
I20260926 07:02:12.895202 9690 raft_consensus.cc:2454] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:12.895303 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2c1619d0644aaab948de9b94125a97" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060" is_pre_election: true
I20260926 07:02:12.895435 9822 raft_consensus.cc:2454] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:12.895457 9484 leader_election.cc:302] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:12.895582 9912 raft_consensus.cc:2781] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:12.895622 9912 raft_consensus.cc:482] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:12.895642 9912 raft_consensus.cc:3037] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.896502 9912 raft_consensus.cc:504] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.896651 9912 leader_election.cc:288] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Requested vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:12.896896 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2c1619d0644aaab948de9b94125a97" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060"
I20260926 07:02:12.896972 9822 raft_consensus.cc:3037] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.897006 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2c1619d0644aaab948de9b94125a97" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170"
I20260926 07:02:12.897087 9690 raft_consensus.cc:3037] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.897636 9822 raft_consensus.cc:2454] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:12.897859 9690 raft_consensus.cc:2454] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:12.898029 9484 leader_election.cc:302] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:12.898165 9912 raft_consensus.cc:2781] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:12.898288 9912 raft_consensus.cc:686] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Becoming Leader. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Running, Role: LEADER
I20260926 07:02:12.898356 9912 consensus_queue.cc:237] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.899063 9392 catalog_manager.cc:5495] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6861959438e74547a2e40553b3e2924a (127.1.217.65). New cstate: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } health_report { overall_health: UNKNOWN } } }
W20260926 07:02:12.903328 9877 tablet.cc:2369] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:02:12 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406132, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.126@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:12.924850 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.917556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45407 (local address 127.0.0.1:41152)
0926 07:02:12.917738 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.917752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.917813 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.917956 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.917957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.918243 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:12.918295 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.918297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.919207 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.919208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.919791 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.919792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.920220 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.920220 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.920261 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.920873 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:12.920877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:12.922163 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:12.923656 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.923662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.923664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.923856 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.924092 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:12.924092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:12.924093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:12.924150 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:12.924443 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:12.924449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:12.924492 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.924557 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.924650 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":110,"threads_started":1}
I20260926 07:02:12.924853 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.917606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.1.217.126:45407)
0926 07:02:12.917673 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.917675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.917817 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:02:12.917876 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.917877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.917888 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.917943 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:12.918362 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.919134 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.919866 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.920139 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.920206 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.922250 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:12.922251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:12.922251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:12.922258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:12.923579 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.923944 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.923946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.923946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.924019 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:12.924228 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:12.924229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:12.924230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:12.924345 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:12.924419 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.924598 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.924699 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":40}
I20260926 07:02:12.926772 9392 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:41152:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 07:02:12.926944 9392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:12.928501 9810 tablet_service.cc:1467] Processing CreateTablet for tablet 13a529d3a1ea4bce8541b33fc8121b9a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3fad9ab537704be98e184a78bd9d415e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:02:12.928508 9669 tablet_service.cc:1467] Processing CreateTablet for tablet 13a529d3a1ea4bce8541b33fc8121b9a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3fad9ab537704be98e184a78bd9d415e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:02:12.928659 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a529d3a1ea4bce8541b33fc8121b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.928673 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a529d3a1ea4bce8541b33fc8121b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.929780 9904 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: Bootstrap starting.
I20260926 07:02:12.929811 9905 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: Bootstrap starting.
I20260926 07:02:12.929909 9530 tablet_service.cc:1467] Processing CreateTablet for tablet 13a529d3a1ea4bce8541b33fc8121b9a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3fad9ab537704be98e184a78bd9d415e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 07:02:12.930044 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13a529d3a1ea4bce8541b33fc8121b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:12.930140 9904 tablet_bootstrap.cc:654] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.930227 9905 tablet_bootstrap.cc:654] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.930756 9904 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: No bootstrap required, opened a new log
I20260926 07:02:12.930815 9904 ts_tablet_manager.cc:1397] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:12.930910 9905 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: No bootstrap required, opened a new log
I20260926 07:02:12.930935 9904 raft_consensus.cc:348] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.930981 9905 ts_tablet_manager.cc:1397] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.931015 9904 raft_consensus.cc:374] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.931042 9904 raft_consensus.cc:729] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.931093 9904 consensus_queue.cc:260] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.931182 9904 ts_tablet_manager.cc:1428] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:12.931275 9905 raft_consensus.cc:348] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.931274 9925 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.931370 9905 raft_consensus.cc:374] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.931373 9925 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.931399 9905 raft_consensus.cc:729] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e342bda0d40b8b086a710bbb6e170, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.931437 9905 consensus_queue.cc:260] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.931533 9905 ts_tablet_manager.cc:1428] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.931579 9906 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: Bootstrap starting.
I20260926 07:02:12.931584 9911 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.931651 9911 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.932000 9906 tablet_bootstrap.cc:654] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:12.932580 9906 tablet_bootstrap.cc:492] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: No bootstrap required, opened a new log
I20260926 07:02:12.932617 9906 ts_tablet_manager.cc:1397] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:12.932719 9906 raft_consensus.cc:348] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.932816 9906 raft_consensus.cc:374] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:12.932843 9906 raft_consensus.cc:729] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Initialized, Role: FOLLOWER
I20260926 07:02:12.932910 9906 consensus_queue.cc:260] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.933007 9906 ts_tablet_manager.cc:1428] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:12.933041 9912 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.933106 9912 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.973793 9912 raft_consensus.cc:482] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:12.973889 9912 raft_consensus.cc:504] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.974038 9912 leader_election.cc:288] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:12.974251 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a529d3a1ea4bce8541b33fc8121b9a" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170" is_pre_election: true
I20260926 07:02:12.974264 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a529d3a1ea4bce8541b33fc8121b9a" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060" is_pre_election: true
I20260926 07:02:12.974359 9690 raft_consensus.cc:2454] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:12.974359 9822 raft_consensus.cc:2454] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:12.974531 9483 leader_election.cc:302] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abe7435fac44c9d86ad86dfa15b2060, 6861959438e74547a2e40553b3e2924a; no voters:
I20260926 07:02:12.974635 9912 raft_consensus.cc:2781] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:12.974680 9912 raft_consensus.cc:482] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:12.974707 9912 raft_consensus.cc:3037] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.975334 9912 raft_consensus.cc:504] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.975482 9912 leader_election.cc:288] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Requested vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:12.975611 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a529d3a1ea4bce8541b33fc8121b9a" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170"
I20260926 07:02:12.975711 9690 raft_consensus.cc:3037] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.975703 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13a529d3a1ea4bce8541b33fc8121b9a" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060"
I20260926 07:02:12.975800 9822 raft_consensus.cc:3037] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:12.976372 9690 raft_consensus.cc:2454] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:12.976444 9822 raft_consensus.cc:2454] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:12.976574 9484 leader_election.cc:302] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:12.976670 9912 raft_consensus.cc:2781] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:12.976734 9912 raft_consensus.cc:686] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Becoming Leader. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Running, Role: LEADER
I20260926 07:02:12.976815 9912 consensus_queue.cc:237] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:12.977180 9935 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: TxnStatusTablet state changed. Reason: New leader 6861959438e74547a2e40553b3e2924a. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.977331 9935 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:02:12.977525 9947 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 07:02:12.977478 9393 catalog_manager.cc:5495] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6861959438e74547a2e40553b3e2924a (127.1.217.65). New cstate: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260926 07:02:12.977617 9947 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 07:02:12.991442 9942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.988627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41239 (local address 127.0.0.1:46904)
0926 07:02:12.988699 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:12.988712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:12.988805 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:12.988990 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:12.988991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:12.988992 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:02:12.989039 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.989041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.989968 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.989968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.990521 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:12.990523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.990981 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.990981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:12.991026 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.991039 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:02:12.991311 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:02:12.991317 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:02:12.991350 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 07:02:12.991385 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:02:12.991458 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.988708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46904 (local address 127.1.217.65:41239)
0926 07:02:12.988749 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.988751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.988760 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:12.988888 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.988888 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.988907 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.988981 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:12.989085 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.989870 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.990602 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.990906 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.990976 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.991129 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:12.991250 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:12.991321 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.991358 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.991378 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:12.992681 9690 raft_consensus.cc:1261] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:12.992774 9822 raft_consensus.cc:1261] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:12.993039 9912 consensus_queue.cc:1035] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:12.993151 9935 consensus_queue.cc:1035] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:12.993775 9925 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: TxnStatusTablet state changed. Reason: New leader 6861959438e74547a2e40553b3e2924a. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.993867 9925 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.993881 9911 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: TxnStatusTablet state changed. Reason: New leader 6861959438e74547a2e40553b3e2924a. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.993957 9911 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.994534 9912 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.994619 9912 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:02:12.994616 9911 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.994685 9911 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P b40e342bda0d40b8b086a710bbb6e170: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.994679 9918 mvcc.cc:205] Tried to move back new op lower bound from 7333503520736129024 to 7333503520673456128. Current Snapshot: MvccSnapshot[applied={T|T < 7333503520736129024}]
I20260926 07:02:12.995172 9935 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.995249 9935 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 6861959438e74547a2e40553b3e2924a: This TxnStatusTablet replica's current role is: LEADER
I20260926 07:02:12.995679 9925 tablet_replica.cc:439] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } } }
I20260926 07:02:12.995800 9925 tablet_replica.cc:442] T 13a529d3a1ea4bce8541b33fc8121b9a P 1abe7435fac44c9d86ad86dfa15b2060: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 07:02:12.999061 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:12.996542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.1.217.65:41239)
0926 07:02:12.996577 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:12.996579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:12.996611 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:02:12.996686 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:12.996686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:12.996696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:12.996749 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:12.996900 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.997625 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.998332 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:12.998682 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:12.998753 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:12.998855 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:12.998888 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:12.998935 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:12.998978 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 07:02:12.998994 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:13.004036 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:13.001469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:48423 (local address 127.1.217.65:41239)
0926 07:02:13.001513 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:13.001515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:13.001659 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:02:13.001713 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:13.001714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:13.001721 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:13.001766 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:13.001876 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.002296 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.003398 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.003606 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.003659 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:13.003834 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:13.003857 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:13.003890 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:13.003941 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:02:13.003958 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:13.004130 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:13.001388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41239 (local address 127.1.217.65:48423)
0926 07:02:13.001589 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:13.001600 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:13.001657 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:13.001770 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:13.001771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:13.001772 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 07:02:13.001823 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:13.001824 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.002362 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.002363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:13.003299 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:13.003301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.003672 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.003673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:13.003740 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:13.003754 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 07:02:13.003891 (+ 137us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 07:02:13.003892 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 07:02:13.003940 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:02:13.003990 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260926 07:02:13.006106 9690 raft_consensus.cc:1261] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.006135 9822 raft_consensus.cc:1261] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.006306 9955 consensus_queue.cc:1035] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.006445 9912 consensus_queue.cc:1035] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.018121 9392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.018234 9392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.018780 9392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Sending DeleteTablet for 3 replicas of tablet 5e2c1619d0644aaab948de9b94125a97
I20260926 07:02:13.019037 9810 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2c1619d0644aaab948de9b94125a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:50656
I20260926 07:02:13.019135 9928 tablet_replica.cc:330] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: stopping tablet replica
I20260926 07:02:13.019121 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2c1619d0644aaab948de9b94125a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33294
I20260926 07:02:13.019187 9928 raft_consensus.cc:2229] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.019203 9927 tablet_replica.cc:330] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: stopping tablet replica
I20260926 07:02:13.019222 9928 raft_consensus.cc:2258] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.019246 9927 raft_consensus.cc:2229] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.019276 9927 raft_consensus.cc:2258] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.019551 9529 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2c1619d0644aaab948de9b94125a97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:46894
I20260926 07:02:13.019595 9927 ts_tablet_manager.cc:1905] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.019636 9929 tablet_replica.cc:330] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: stopping tablet replica
I20260926 07:02:13.019696 9929 raft_consensus.cc:2229] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:13.019773 9929 raft_consensus.cc:2258] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.020064 9929 ts_tablet_manager.cc:1905] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.021173 9927 ts_tablet_manager.cc:1918] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:02:13.021220 9927 log.cc:1196] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/5e2c1619d0644aaab948de9b94125a97
I20260926 07:02:13.021423 9927 ts_tablet_manager.cc:1939] T 5e2c1619d0644aaab948de9b94125a97 P b40e342bda0d40b8b086a710bbb6e170: Deleting consensus metadata
I20260926 07:02:13.021770 9929 ts_tablet_manager.cc:1918] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:02:13.021828 9929 log.cc:1196] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/5e2c1619d0644aaab948de9b94125a97
I20260926 07:02:13.022039 9929 ts_tablet_manager.cc:1939] T 5e2c1619d0644aaab948de9b94125a97 P 6861959438e74547a2e40553b3e2924a: Deleting consensus metadata
I20260926 07:02:13.022435 9377 catalog_manager.cc:4841] TS 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239): tablet 5e2c1619d0644aaab948de9b94125a97 (table test-table [id=6bad7d2e323d4eefb24d691480d7210d]) successfully deleted
I20260926 07:02:13.022619 9928 ts_tablet_manager.cc:1905] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.023072 9380 catalog_manager.cc:4841] TS b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309): tablet 5e2c1619d0644aaab948de9b94125a97 (table test-table [id=6bad7d2e323d4eefb24d691480d7210d]) successfully deleted
I20260926 07:02:13.024009 9928 ts_tablet_manager.cc:1918] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:02:13.024084 9928 log.cc:1196] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/5e2c1619d0644aaab948de9b94125a97
I20260926 07:02:13.024307 9928 ts_tablet_manager.cc:1939] T 5e2c1619d0644aaab948de9b94125a97 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting consensus metadata
I20260926 07:02:13.024624 9379 catalog_manager.cc:4841] TS 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629): tablet 5e2c1619d0644aaab948de9b94125a97 (table test-table [id=6bad7d2e323d4eefb24d691480d7210d]) successfully deleted
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:02:13.047241 9443 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:13.041589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.1.217.126:45407)
0926 07:02:13.041656 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:13.041658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:13.041755 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:02:13.041806 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:13.041807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:13.041819 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:13.041890 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:13.042454 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.043239 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.043985 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.044293 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.044365 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:13.047173 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41166: BlockingRecv error: recv got EOF from 127.0.0.1:41166 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM
I20260926 07:02:13.055769 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:13.048077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.1.217.126:45407)
0926 07:02:13.048112 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:13.048114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:13.048266 (+ 152us) server_negotiation.cc:407] Connection header received
0926 07:02:13.048298 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:13.048299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:13.048310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:13.048367 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:13.048993 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.049609 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.050578 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.050751 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.050806 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:13.053142 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:13.053143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:13.053143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:13.053151 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:13.054568 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:13.054927 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:13.054928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:13.054929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:13.054993 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:13.055222 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:13.055223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:13.055223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:13.055334 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:13.055439 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:13.055552 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 07:02:13.055666 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:13.057356 9392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:13.057508 9392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:13.058835 9810 tablet_service.cc:1467] Processing CreateTablet for tablet 97efb350e161485a9426a840ab6f16d8 (DEFAULT_TABLE table=test-table [id=5f8468584553456faa7939ee4f941205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.058846 9529 tablet_service.cc:1467] Processing CreateTablet for tablet 97efb350e161485a9426a840ab6f16d8 (DEFAULT_TABLE table=test-table [id=5f8468584553456faa7939ee4f941205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.058867 9669 tablet_service.cc:1467] Processing CreateTablet for tablet 97efb350e161485a9426a840ab6f16d8 (DEFAULT_TABLE table=test-table [id=5f8468584553456faa7939ee4f941205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.058970 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97efb350e161485a9426a840ab6f16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.058970 9529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97efb350e161485a9426a840ab6f16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.058995 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97efb350e161485a9426a840ab6f16d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.060276 9904 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Bootstrap starting.
I20260926 07:02:13.060281 9906 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Bootstrap starting.
I20260926 07:02:13.060315 9905 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Bootstrap starting.
I20260926 07:02:13.060562 9906 tablet_bootstrap.cc:654] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.060636 9904 tablet_bootstrap.cc:654] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.060686 9905 tablet_bootstrap.cc:654] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.061165 9906 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: No bootstrap required, opened a new log
I20260926 07:02:13.061223 9906 ts_tablet_manager.cc:1397] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:13.061394 9906 raft_consensus.cc:348] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.061460 9904 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: No bootstrap required, opened a new log
I20260926 07:02:13.061481 9906 raft_consensus.cc:374] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.061501 9904 ts_tablet_manager.cc:1397] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:13.061514 9906 raft_consensus.cc:729] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.061571 9906 consensus_queue.cc:260] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.061601 9904 raft_consensus.cc:348] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.061668 9904 raft_consensus.cc:374] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.061667 9906 ts_tablet_manager.cc:1428] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:13.061697 9904 raft_consensus.cc:729] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.061748 9904 consensus_queue.cc:260] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.061836 9904 ts_tablet_manager.cc:1428] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:13.065209 9905 tablet_bootstrap.cc:492] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: No bootstrap required, opened a new log
I20260926 07:02:13.065268 9905 ts_tablet_manager.cc:1397] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 07:02:13.065408 9905 raft_consensus.cc:348] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.065483 9905 raft_consensus.cc:374] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.065515 9905 raft_consensus.cc:729] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e342bda0d40b8b086a710bbb6e170, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.065567 9905 consensus_queue.cc:260] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.065663 9905 ts_tablet_manager.cc:1428] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:13.289435 9955 raft_consensus.cc:482] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:13.289551 9955 raft_consensus.cc:504] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.289740 9955 leader_election.cc:288] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:13.289899 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97efb350e161485a9426a840ab6f16d8" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170" is_pre_election: true
I20260926 07:02:13.290030 9690 raft_consensus.cc:2454] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:13.289984 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97efb350e161485a9426a840ab6f16d8" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060" is_pre_election: true
I20260926 07:02:13.290078 9822 raft_consensus.cc:2454] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:13.290252 9484 leader_election.cc:302] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:13.290385 9955 raft_consensus.cc:2781] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:13.290426 9955 raft_consensus.cc:482] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:13.290472 9955 raft_consensus.cc:3037] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.291234 9955 raft_consensus.cc:504] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.291385 9955 leader_election.cc:288] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Requested vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:13.291538 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97efb350e161485a9426a840ab6f16d8" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170"
I20260926 07:02:13.291553 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97efb350e161485a9426a840ab6f16d8" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060"
I20260926 07:02:13.291621 9690 raft_consensus.cc:3037] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.291621 9822 raft_consensus.cc:3037] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.292392 9822 raft_consensus.cc:2454] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:13.292405 9690 raft_consensus.cc:2454] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:13.292622 9483 leader_election.cc:302] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1abe7435fac44c9d86ad86dfa15b2060, 6861959438e74547a2e40553b3e2924a; no voters:
I20260926 07:02:13.292737 9955 raft_consensus.cc:2781] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:13.292855 9955 raft_consensus.cc:686] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Becoming Leader. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Running, Role: LEADER
I20260926 07:02:13.292917 9955 consensus_queue.cc:237] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.293556 9392 catalog_manager.cc:5495] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6861959438e74547a2e40553b3e2924a (127.1.217.65). New cstate: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260926 07:02:13.347368 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:13.344319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46918 (local address 127.1.217.65:41239)
0926 07:02:13.344366 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:13.344369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:13.344377 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:13.344403 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:13.344404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:13.344435 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:13.344512 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:13.344671 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.345474 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.346433 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:13.346726 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:13.346803 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:13.347039 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:13.347079 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:13.347124 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:13.347240 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 07:02:13.347262 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:02:13.348130 9822 raft_consensus.cc:1261] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.348183 9690 raft_consensus.cc:1261] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.348366 9935 consensus_queue.cc:1035] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.348466 9955 consensus_queue.cc:1035] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.351449 9392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.351536 9392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.352165 9392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Sending DeleteTablet for 3 replicas of tablet 97efb350e161485a9426a840ab6f16d8
I20260926 07:02:13.352422 9529 tablet_service.cc:1514] Processing DeleteTablet for tablet 97efb350e161485a9426a840ab6f16d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:46894
I20260926 07:02:13.352450 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet 97efb350e161485a9426a840ab6f16d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33294
I20260926 07:02:13.352520 9929 tablet_replica.cc:330] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: stopping tablet replica
I20260926 07:02:13.352509 9810 tablet_service.cc:1514] Processing DeleteTablet for tablet 97efb350e161485a9426a840ab6f16d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:50656
I20260926 07:02:13.352568 9929 raft_consensus.cc:2229] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:13.352576 9927 tablet_replica.cc:330] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: stopping tablet replica
I20260926 07:02:13.352586 9928 tablet_replica.cc:330] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: stopping tablet replica
I20260926 07:02:13.352620 9929 raft_consensus.cc:2258] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.352628 9927 raft_consensus.cc:2229] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.352630 9928 raft_consensus.cc:2229] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.352680 9927 raft_consensus.cc:2258] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.352684 9928 raft_consensus.cc:2258] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.353035 9927 ts_tablet_manager.cc:1905] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.353057 9929 ts_tablet_manager.cc:1905] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.353060 9928 ts_tablet_manager.cc:1905] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.354602 9929 ts_tablet_manager.cc:1918] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:13.354662 9929 log.cc:1196] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/97efb350e161485a9426a840ab6f16d8
I20260926 07:02:13.354683 9927 ts_tablet_manager.cc:1918] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:13.354683 9928 ts_tablet_manager.cc:1918] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:13.354741 9927 log.cc:1196] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/97efb350e161485a9426a840ab6f16d8
I20260926 07:02:13.354741 9928 log.cc:1196] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/97efb350e161485a9426a840ab6f16d8
I20260926 07:02:13.354652 9392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:13.354810 9392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:13.354887 9929 ts_tablet_manager.cc:1939] T 97efb350e161485a9426a840ab6f16d8 P 6861959438e74547a2e40553b3e2924a: Deleting consensus metadata
I20260926 07:02:13.354969 9928 ts_tablet_manager.cc:1939] T 97efb350e161485a9426a840ab6f16d8 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting consensus metadata
I20260926 07:02:13.354969 9927 ts_tablet_manager.cc:1939] T 97efb350e161485a9426a840ab6f16d8 P b40e342bda0d40b8b086a710bbb6e170: Deleting consensus metadata
I20260926 07:02:13.355306 9379 catalog_manager.cc:4841] TS 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629): tablet 97efb350e161485a9426a840ab6f16d8 (table test-table [id=5f8468584553456faa7939ee4f941205]) successfully deleted
I20260926 07:02:13.355301 9380 catalog_manager.cc:4841] TS b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309): tablet 97efb350e161485a9426a840ab6f16d8 (table test-table [id=5f8468584553456faa7939ee4f941205]) successfully deleted
I20260926 07:02:13.355433 9377 catalog_manager.cc:4841] TS 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239): tablet 97efb350e161485a9426a840ab6f16d8 (table test-table [id=5f8468584553456faa7939ee4f941205]) successfully deleted
I20260926 07:02:13.356288 9669 tablet_service.cc:1467] Processing CreateTablet for tablet a35b9fbdbed3484387a22590a8c9b798 (DEFAULT_TABLE table=test-table [id=2f57264895e24b0c8715ddac50215f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.356295 9810 tablet_service.cc:1467] Processing CreateTablet for tablet a35b9fbdbed3484387a22590a8c9b798 (DEFAULT_TABLE table=test-table [id=2f57264895e24b0c8715ddac50215f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.356302 9529 tablet_service.cc:1467] Processing CreateTablet for tablet a35b9fbdbed3484387a22590a8c9b798 (DEFAULT_TABLE table=test-table [id=2f57264895e24b0c8715ddac50215f3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:13.356423 9529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35b9fbdbed3484387a22590a8c9b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.356424 9669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35b9fbdbed3484387a22590a8c9b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.356423 9810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a35b9fbdbed3484387a22590a8c9b798. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:13.357424 9905 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Bootstrap starting.
I20260926 07:02:13.357647 9906 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: Bootstrap starting.
I20260926 07:02:13.357667 9904 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Bootstrap starting.
I20260926 07:02:13.357878 9905 tablet_bootstrap.cc:654] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.357904 9906 tablet_bootstrap.cc:654] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.358003 9904 tablet_bootstrap.cc:654] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:13.358404 9906 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: No bootstrap required, opened a new log
I20260926 07:02:13.358484 9906 ts_tablet_manager.cc:1397] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:02:13.358531 9905 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: No bootstrap required, opened a new log
I20260926 07:02:13.358583 9905 ts_tablet_manager.cc:1397] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:13.358604 9904 tablet_bootstrap.cc:492] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: No bootstrap required, opened a new log
I20260926 07:02:13.358651 9904 ts_tablet_manager.cc:1397] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:13.358609 9906 raft_consensus.cc:348] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358688 9906 raft_consensus.cc:374] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.358718 9906 raft_consensus.cc:729] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.358759 9906 consensus_queue.cc:260] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358759 9904 raft_consensus.cc:348] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358747 9905 raft_consensus.cc:348] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358819 9904 raft_consensus.cc:374] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.358826 9905 raft_consensus.cc:374] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:13.358844 9904 raft_consensus.cc:729] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1abe7435fac44c9d86ad86dfa15b2060, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.358847 9905 raft_consensus.cc:729] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40e342bda0d40b8b086a710bbb6e170, State: Initialized, Role: FOLLOWER
I20260926 07:02:13.358860 9906 ts_tablet_manager.cc:1428] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:13.358886 9904 consensus_queue.cc:260] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358892 9905 consensus_queue.cc:260] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.358980 9905 ts_tablet_manager.cc:1428] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:13.358983 9904 ts_tablet_manager.cc:1428] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:13.583395 9955 raft_consensus.cc:482] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:13.583547 9955 raft_consensus.cc:504] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.583761 9955 leader_election.cc:288] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:13.583989 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35b9fbdbed3484387a22590a8c9b798" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170" is_pre_election: true
I20260926 07:02:13.584085 9690 raft_consensus.cc:2454] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:13.584270 9484 leader_election.cc:302] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:13.584374 9955 raft_consensus.cc:2781] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:13.584415 9955 raft_consensus.cc:482] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:13.584431 9955 raft_consensus.cc:3037] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.584503 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35b9fbdbed3484387a22590a8c9b798" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060" is_pre_election: true
I20260926 07:02:13.584589 9822 raft_consensus.cc:2454] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 0.
I20260926 07:02:13.585381 9955 raft_consensus.cc:504] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.585522 9955 leader_election.cc:288] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Requested vote from peers b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309), 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629)
I20260926 07:02:13.585652 9690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35b9fbdbed3484387a22590a8c9b798" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b40e342bda0d40b8b086a710bbb6e170"
I20260926 07:02:13.585668 9822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a35b9fbdbed3484387a22590a8c9b798" candidate_uuid: "6861959438e74547a2e40553b3e2924a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1abe7435fac44c9d86ad86dfa15b2060"
I20260926 07:02:13.585731 9690 raft_consensus.cc:3037] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.585740 9822 raft_consensus.cc:3037] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:13.586369 9690 raft_consensus.cc:2454] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:13.586380 9822 raft_consensus.cc:2454] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6861959438e74547a2e40553b3e2924a in term 1.
I20260926 07:02:13.586577 9484 leader_election.cc:302] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6861959438e74547a2e40553b3e2924a, b40e342bda0d40b8b086a710bbb6e170; no voters:
I20260926 07:02:13.586694 9955 raft_consensus.cc:2781] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:13.586761 9955 raft_consensus.cc:686] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Becoming Leader. State: Replica: 6861959438e74547a2e40553b3e2924a, State: Running, Role: LEADER
I20260926 07:02:13.586822 9955 consensus_queue.cc:237] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } }
I20260926 07:02:13.587402 9392 catalog_manager.cc:5495] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6861959438e74547a2e40553b3e2924a (127.1.217.65). New cstate: current_term: 1 leader_uuid: "6861959438e74547a2e40553b3e2924a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6861959438e74547a2e40553b3e2924a" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 } health_report { overall_health: UNKNOWN } } }
I20260926 07:02:13.646521 9822 raft_consensus.cc:1261] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.646709 9690 raft_consensus.cc:1261] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Refusing update from remote peer 6861959438e74547a2e40553b3e2924a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:13.646744 9935 consensus_queue.cc:1035] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1abe7435fac44c9d86ad86dfa15b2060" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.646883 9955 consensus_queue.cc:1035] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b40e342bda0d40b8b086a710bbb6e170" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 41309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:13.659282 9392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.659374 9392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:13.659989 9392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae50e1b7571741e1b158b68b17bd7f4d: Sending DeleteTablet for 3 replicas of tablet a35b9fbdbed3484387a22590a8c9b798
I20260926 07:02:13.660218 9530 tablet_service.cc:1514] Processing DeleteTablet for tablet a35b9fbdbed3484387a22590a8c9b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:46894
I20260926 07:02:13.660259 9669 tablet_service.cc:1514] Processing DeleteTablet for tablet a35b9fbdbed3484387a22590a8c9b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33294
I20260926 07:02:13.660317 9929 tablet_replica.cc:330] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a: stopping tablet replica
I20260926 07:02:13.660362 9929 raft_consensus.cc:2229] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:13.660358 9927 tablet_replica.cc:330] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: stopping tablet replica
I20260926 07:02:13.660411 9927 raft_consensus.cc:2229] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.660425 9929 raft_consensus.cc:2258] T a35b9fbdbed3484387a22590a8c9b798 P 6861959438e74547a2e40553b3e2924a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.660454 9927 raft_consensus.cc:2258] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.660493 9810 tablet_service.cc:1514] Processing DeleteTablet for tablet a35b9fbdbed3484387a22590a8c9b798 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:13 UTC) from {username='oryx', principal='oryx/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:50656
I20260926 07:02:13.660586 9928 tablet_replica.cc:330] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: stopping tablet replica
I20260926 07:02:13.660647 9928 raft_consensus.cc:2229] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:13.660682 9928 raft_consensus.cc:2258] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:13.660789 9927 ts_tablet_manager.cc:1905] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.660959 9928 ts_tablet_manager.cc:1905] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:13.661069 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 9465
I20260926 07:02:13.662709 9928 ts_tablet_manager.cc:1918] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:02:13.662751 9927 ts_tablet_manager.cc:1918] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 07:02:13.662775 9928 log.cc:1196] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/a35b9fbdbed3484387a22590a8c9b798
I20260926 07:02:13.662798 9927 log.cc:1196] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/a35b9fbdbed3484387a22590a8c9b798
I20260926 07:02:13.662997 9928 ts_tablet_manager.cc:1939] T a35b9fbdbed3484387a22590a8c9b798 P 1abe7435fac44c9d86ad86dfa15b2060: Deleting consensus metadata
I20260926 07:02:13.663002 9927 ts_tablet_manager.cc:1939] T a35b9fbdbed3484387a22590a8c9b798 P b40e342bda0d40b8b086a710bbb6e170: Deleting consensus metadata
I20260926 07:02:13.663349 9379 catalog_manager.cc:4841] TS 1abe7435fac44c9d86ad86dfa15b2060 (127.1.217.67:34629): tablet a35b9fbdbed3484387a22590a8c9b798 (table test-table [id=2f57264895e24b0c8715ddac50215f3b]) successfully deleted
I20260926 07:02:13.663363 9380 catalog_manager.cc:4841] TS b40e342bda0d40b8b086a710bbb6e170 (127.1.217.66:41309): tablet a35b9fbdbed3484387a22590a8c9b798 (table test-table [id=2f57264895e24b0c8715ddac50215f3b]) successfully deleted
W20260926 07:02:13.668951 9377 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:41239 (error 108)
I20260926 07:02:13.669225 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 9605
I20260926 07:02:13.669828 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:13.669316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41239 (local address 127.0.0.1:46920)
0926 07:02:13.669751 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:13.669789 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":35,"threads_started":1}
W20260926 07:02:13.669929 9377 catalog_manager.cc:4584] TS 6861959438e74547a2e40553b3e2924a (127.1.217.65:41239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a35b9fbdbed3484387a22590a8c9b798: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41239: connect: Connection refused (error 111)
I20260926 07:02:13.674245 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 9745
I20260926 07:02:13.679682 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 9361
2026-09-26T07:02:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2813 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9990](info): set up 2 sockets
Sep 26 07:02:13 dist-test-slave-t1mc krb5kdc[9990](info): commencing operation
krb5kdc: starting...
W20260926 07:02:14.717417 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:14 dist-test-slave-t1mc krb5kdc[9990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:02:14.835002 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33521
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:14.914600 10003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:14.914815 10003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:14.914839 10003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:14.915948 10003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:14.915984 10003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:14.915998 10003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:14.916011 10003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:14.916024 10003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:14.917591 10003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:14.917805 10003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:14.917975 10003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:14.922776 10009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:14.922832 10011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:14.922927 10003 server_base.cc:1034] running on GCE node
W20260926 07:02:14.922830 10008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:14.923296 10003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:14.923616 10003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:14.924825 10003 hybrid_clock.cc:648] HybridClock initialized: now 1790406134924795 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T07:02:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1251 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:14 dist-test-slave-t1mc krb5kdc[10016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:14 dist-test-slave-t1mc krb5kdc[10016](info): set up 2 sockets
Sep 26 07:02:14 dist-test-slave-t1mc krb5kdc[10016](info): commencing operation
krb5kdc: starting...
W20260926 07:02:15.973228 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:02:16.091846 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42403
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:16.170696 10029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.170931 10029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.170967 10029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.172083 10029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:16.172151 10029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.172174 10029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.172195 10029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:16.172227 10029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:16.173707 10029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.173977 10029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.174192 10029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.178830 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.178869 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.178875 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.179154 10029 server_base.cc:1034] running on GCE node
I20260926 07:02:16.179404 10029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.179761 10029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.180908 10029 hybrid_clock.cc:648] HybridClock initialized: now 1790406136180893 us; error 30 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.183202 10029 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:02:16.183627 10029 webserver.cc:460] Webserver started at http://127.1.217.126:38247/ using document root <none> and password file <none>
I20260926 07:02:16.183840 10029 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.183881 10029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.183966 10029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:16.184981 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "9abeba26db814eddb59f9fe474aa1317"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "d18045f48c6a8b7c80525d6b93e17c52"
server_key_iv: "6591417137d34fb5355488ac314d969e"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.185257 10029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "9abeba26db814eddb59f9fe474aa1317"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "d18045f48c6a8b7c80525d6b93e17c52"
server_key_iv: "6591417137d34fb5355488ac314d969e"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.186436 10029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:16.187054 10044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.187213 10029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:16.187309 10029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "9abeba26db814eddb59f9fe474aa1317"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "d18045f48c6a8b7c80525d6b93e17c52"
server_key_iv: "6591417137d34fb5355488ac314d969e"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.187391 10029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:16.198036 10029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:16.201016 10029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:16.201151 10029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:16.205796 10029 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42403
I20260926 07:02:16.205825 10096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42403 every 8 connection(s)
I20260926 07:02:16.206176 10029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:16.206387 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10029
I20260926 07:02:16.206518 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:16.206753 1893 external_mini_cluster.cc:1428] Setting key fbaa6fdea640a156aa787741b9cb5678
I20260926 07:02:16.207181 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:16.210361 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Bootstrap starting.
I20260926 07:02:16.211016 10097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:16.211310 10097 log.cc:824] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:16.212003 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: No bootstrap required, opened a new log
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:16.213939 10097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.214092 10097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:16.214138 10097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abeba26db814eddb59f9fe474aa1317, State: Initialized, Role: FOLLOWER
I20260926 07:02:16.214301 10097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.214395 10097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:16.214439 10097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:16.214478 10097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:16.214980 10097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.215063 10097 leader_election.cc:302] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abeba26db814eddb59f9fe474aa1317; no voters:
I20260926 07:02:16.215234 10097 leader_election.cc:288] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:16.215277 10102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:16.215581 10102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 LEADER]: Becoming Leader. State: Replica: 9abeba26db814eddb59f9fe474aa1317, State: Running, Role: LEADER
I20260926 07:02:16.215710 10097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:16.215716 10102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.216048 10102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abeba26db814eddb59f9fe474aa1317. Latest consensus state: current_term: 1 leader_uuid: "9abeba26db814eddb59f9fe474aa1317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } } }
I20260926 07:02:16.216104 10102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:16.216271 10104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9abeba26db814eddb59f9fe474aa1317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } } }
I20260926 07:02:16.216325 10104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: This master's current role is: LEADER
W20260926 07:02:16.217793 10118 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 07:02:16.217994 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.207272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.1.217.126:42403)
0926 07:02:16.207441 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:16.207449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:16.207530 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:02:16.208302 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:16.208309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:16.208358 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:16.208470 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:16.209273 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.210134 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.211196 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.211625 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.211752 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.214134 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:16.214138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:16.214138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:16.214151 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:16.216071 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.216460 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.216463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.216464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.216573 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.216761 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.216761 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.216780 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.217091 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:16.217256 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:16.217624 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 07:02:16.217794 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
W20260926 07:02:16.218235 10118 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:02:16.218513 10106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:16.218703 10106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:16.220221 10106 catalog_manager.cc:1347] Generated new cluster ID: b3db420885e8445e8c74c81f5625d456
I20260926 07:02:16.220274 10106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:16.223325 10106 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:16.223826 10106 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:16.226990 10106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Generated new TSK 0
I20260926 07:02:16.227149 10106 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:02:16.261063 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:16.344573 10123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.344780 10123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.344835 10123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.345969 10123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.346007 10123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.346033 10123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:16.347414 10123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.347635 10123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.347877 10123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.353410 10131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.353389 10128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.353403 10129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.353740 10123 server_base.cc:1034] running on GCE node
I20260926 07:02:16.353950 10123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.354202 10123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.355355 10123 hybrid_clock.cc:648] HybridClock initialized: now 1790406136355343 us; error 31 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.357955 10123 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:02:16.358384 10123 webserver.cc:460] Webserver started at http://127.1.217.65:33487/ using document root <none> and password file <none>
I20260926 07:02:16.358601 10123 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.358667 10123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.358809 10123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:16.359834 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "08a0a89a53a64c80b663d2aef66d726a"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "725c1e3b5d077eb5725cf0fb114ea5be"
server_key_iv: "856d1be3c22e81177a9210d8fcca9688"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.360194 10123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "08a0a89a53a64c80b663d2aef66d726a"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "725c1e3b5d077eb5725cf0fb114ea5be"
server_key_iv: "856d1be3c22e81177a9210d8fcca9688"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.361541 10123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:16.362298 10138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.362484 10123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:16.362582 10123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "08a0a89a53a64c80b663d2aef66d726a"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "725c1e3b5d077eb5725cf0fb114ea5be"
server_key_iv: "856d1be3c22e81177a9210d8fcca9688"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.362668 10123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:16.376345 10123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:16.378198 10123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:16.378356 10123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:16.378577 10123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:16.378896 10123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:16.378948 10123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.378983 10123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:16.379009 10123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.384792 10123 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43321
I20260926 07:02:16.384835 10251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43321 every 8 connection(s)
I20260926 07:02:16.385131 10123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:16.385984 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10123
I20260926 07:02:16.386097 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:16.386289 1893 external_mini_cluster.cc:1428] Setting key 58763411772d549f5876dad13b648f94
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
I20260926 07:02:16.395409 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.385747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:59801 (local address 127.1.217.126:42403)
0926 07:02:16.385826 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:16.385828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:16.386043 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:02:16.386726 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:16.386728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:16.386739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:16.386809 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:16.388058 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.388715 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.389680 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.390063 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.390132 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.392679 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:16.392680 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:16.392680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:16.392686 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:16.394076 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.394572 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.394573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.394573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.394633 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.394907 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.394908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.394908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.395002 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:16.395052 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:16.395283 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 07:02:16.395345 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:02:16.395645 10254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.385707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:59801)
0926 07:02:16.385894 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:16.385944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:16.386610 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:16.386860 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:16.386863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:16.387691 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:16.387969 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.387976 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.388834 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.388835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.389595 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.389597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.390145 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.390145 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.390250 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.390767 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:16.390775 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:16.392588 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:16.394165 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.394171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.394173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.394445 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.394708 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.394709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.394710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.394846 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.395059 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:16.395064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:16.395168 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 07:02:16.395247 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:02:16.395389 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260926 07:02:16.396229 10252 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42403
I20260926 07:02:16.396322 10252 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:16.396458 10252 heartbeater.cc:511] Master 127.1.217.126:42403 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
I20260926 07:02:16.397008 10061 ts_manager.cc:194] Registered new tserver with Master: 08a0a89a53a64c80b663d2aef66d726a (127.1.217.65:43321)
I20260926 07:02:16.397612 10061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:59801
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:02:16.409600 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:16.488502 10257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.488700 10257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.488736 10257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.490026 10257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.490067 10257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.490094 10257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:16.491580 10257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.491829 10257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.492057 10257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.496928 10265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.496958 10262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.496978 10257 server_base.cc:1034] running on GCE node
W20260926 07:02:16.496948 10263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.497512 10257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.497787 10257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.498951 10257 hybrid_clock.cc:648] HybridClock initialized: now 1790406136498936 us; error 48 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.501464 10257 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:02:16.501940 10257 webserver.cc:460] Webserver started at http://127.1.217.66:35937/ using document root <none> and password file <none>
I20260926 07:02:16.502166 10257 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.502242 10257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.502393 10257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:16.503443 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "830602a0468b4cfdaea8ea074d0fde8b"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "33c69882cd628f4b05e23354691fd795"
server_key_iv: "ce75022df480620a41c3f39b36a32f0b"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.503829 10257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "830602a0468b4cfdaea8ea074d0fde8b"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "33c69882cd628f4b05e23354691fd795"
server_key_iv: "ce75022df480620a41c3f39b36a32f0b"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.505201 10257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:16.506001 10272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.506201 10257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:16.506310 10257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "830602a0468b4cfdaea8ea074d0fde8b"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "33c69882cd628f4b05e23354691fd795"
server_key_iv: "ce75022df480620a41c3f39b36a32f0b"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.506385 10257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:16.520750 10257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:16.522845 10257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:16.523013 10257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:16.523262 10257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:16.523597 10257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:16.523658 10257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.523703 10257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:16.523726 10257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.529678 10257 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:38195
I20260926 07:02:16.529718 10385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:38195 every 8 connection(s)
I20260926 07:02:16.530049 10257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:16.533874 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10257
I20260926 07:02:16.533999 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:16.534195 1893 external_mini_cluster.cc:1428] Setting key 19ecb2a8e748a5612fc8197e4335fdbf
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:16.540011 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.530808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:53709)
0926 07:02:16.530980 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:16.531022 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:16.531506 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:16.531718 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:16.531722 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:16.532336 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:16.532532 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.532537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.533354 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.533355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.534289 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.534293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.534734 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.534735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.534894 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.535493 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:16.535501 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:16.536933 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:16.538686 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.538692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.538694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.538922 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.539137 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.539137 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.539137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.539231 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.539528 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:16.539532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:16.539642 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 07:02:16.539722 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:02:16.539852 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 07:02:16.540004 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.530824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:53709 (local address 127.1.217.126:42403)
0926 07:02:16.530892 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:16.530894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:16.531118 (+ 224us) server_negotiation.cc:407] Connection header received
0926 07:02:16.531588 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:16.531589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:16.531598 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:16.531659 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:16.532646 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.533237 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.534407 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.534653 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.534708 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.537028 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:16.537030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:16.537030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:16.537038 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:16.538605 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.538998 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.539000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.539000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.539065 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.539298 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.539299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.539299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.539458 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:16.539538 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:16.539771 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 07:02:16.539868 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:16.540593 10386 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42403
I20260926 07:02:16.540688 10386 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:16.540896 10386 heartbeater.cc:511] Master 127.1.217.126:42403 requested a full tablet report, sending...
I20260926 07:02:16.541301 10061 ts_manager.cc:194] Registered new tserver with Master: 830602a0468b4cfdaea8ea074d0fde8b (127.1.217.66:38195)
I20260926 07:02:16.541836 10061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:53709
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:02:16.557830 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:16.637948 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.638147 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.638170 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.639307 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.639349 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.639375 10391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:16.640810 10391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.641032 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.641273 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.646024 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.646065 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.646016 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.646215 10391 server_base.cc:1034] running on GCE node
I20260926 07:02:16.646461 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.646697 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.647871 10391 hybrid_clock.cc:648] HybridClock initialized: now 1790406136647851 us; error 34 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.650346 10391 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:02:16.650779 10391 webserver.cc:460] Webserver started at http://127.1.217.67:45093/ using document root <none> and password file <none>
I20260926 07:02:16.651006 10391 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.651072 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.651225 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:16.652315 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "98e24fa85af74417b8915ecda356d6fd"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "b36d01953bc7b85fd9cb5fe35ee7cb68"
server_key_iv: "2ae3db3eed5a33472f38d573089af6e6"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.652670 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "98e24fa85af74417b8915ecda356d6fd"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "b36d01953bc7b85fd9cb5fe35ee7cb68"
server_key_iv: "2ae3db3eed5a33472f38d573089af6e6"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.653981 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:16.654819 10406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.655023 10391 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:16.655128 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "98e24fa85af74417b8915ecda356d6fd"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "b36d01953bc7b85fd9cb5fe35ee7cb68"
server_key_iv: "2ae3db3eed5a33472f38d573089af6e6"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.655198 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:16.669760 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:16.672058 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:16.672225 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:16.672421 10391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:16.672731 10391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:16.672801 10391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.672868 10391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:16.672897 10391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.678406 10391 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:41113
I20260926 07:02:16.678453 10519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:41113 every 8 connection(s)
I20260926 07:02:16.678763 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:16.682391 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10391
I20260926 07:02:16.682478 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:16.682679 1893 external_mini_cluster.cc:1428] Setting key 99472bbf11ed9275f3e175c974cde142
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:16.687330 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.679467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:39295 (local address 127.1.217.126:42403)
0926 07:02:16.679542 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:16.679545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:16.679711 (+ 166us) server_negotiation.cc:407] Connection header received
0926 07:02:16.680181 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:16.680182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:16.680192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:16.680244 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:16.681133 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.681618 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.682645 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.682910 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.682994 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.685127 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:16.685128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:16.685128 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:16.685134 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:16.686214 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.686558 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.686559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.686559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.686606 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:16.686830 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:16.686831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:16.686831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:16.686928 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:16.686977 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:16.687212 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 07:02:16.687269 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:02:16.687482 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:16.679427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:39295)
0926 07:02:16.679591 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:16.679623 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:16.680099 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:16.680297 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:16.680300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:16.680872 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:16.681059 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.681063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.681721 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.681722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.682546 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:16.682549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:16.683001 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:16.683002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:16.683162 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:16.683661 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:16.683669 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:16.685049 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:16.686280 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.686285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.686287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.686494 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.686664 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:16.686664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:16.686665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:16.686777 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:16.686998 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:16.687002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:16.687105 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 07:02:16.687183 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:02:16.687291 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260926 07:02:16.687925 10520 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42403
I20260926 07:02:16.688025 10520 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:16.688192 10520 heartbeater.cc:511] Master 127.1.217.126:42403 requested a full tablet report, sending...
I20260926 07:02:16.688591 10061 ts_manager.cc:194] Registered new tserver with Master: 98e24fa85af74417b8915ecda356d6fd (127.1.217.67:41113)
I20260926 07:02:16.689072 10061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:39295
I20260926 07:02:16.694336 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
Entry for principal oryx/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab.
WARNING: no policy specified for oryx/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
Entry for principal oryx/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab.
WARNING: no policy specified for oryx/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
Entry for principal oryx/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab.
I20260926 07:02:16.759316 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10123
I20260926 07:02:16.763495 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10257
I20260926 07:02:16.767674 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10391
I20260926 07:02:16.771749 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10029
I20260926 07:02:16.776724 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42403
--webserver_interface=127.1.217.126
--webserver_port=38247
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:16.856959 10529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.857153 10529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.857175 10529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.858294 10529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:16.858333 10529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.858347 10529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.858359 10529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:16.858429 10529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:16.859936 10529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.860181 10529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.860399 10529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.865103 10535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.865110 10537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.865110 10534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.865348 10529 server_base.cc:1034] running on GCE node
I20260926 07:02:16.865578 10529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.865840 10529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.867012 10529 hybrid_clock.cc:648] HybridClock initialized: now 1790406136866994 us; error 36 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.869452 10529 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:02:16.869890 10529 webserver.cc:460] Webserver started at http://127.1.217.126:38247/ using document root <none> and password file <none>
I20260926 07:02:16.870108 10529 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.870188 10529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.871474 10529 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:16.872107 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:16.872294 10529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:16.872392 10529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "9abeba26db814eddb59f9fe474aa1317"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "d18045f48c6a8b7c80525d6b93e17c52"
server_key_iv: "6591417137d34fb5355488ac314d969e"
server_key_version: "encryptionkey@0"
I20260926 07:02:16.872714 10529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:16.891012 10529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:16.893123 10529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:16.893239 10529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:16.897133 10529 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42403
I20260926 07:02:16.897236 10596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42403 every 8 connection(s)
I20260926 07:02:16.897571 10529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:16.899009 10597 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:02:16.899652 10597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Bootstrap starting.
I20260926 07:02:16.901289 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10529
I20260926 07:02:16.901657 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:43321
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:42403
--webserver_port=33487
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:02:16.908015 10597 log.cc:824] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:16.909732 10597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:02:16.910022 10597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Bootstrap complete.
I20260926 07:02:16.912201 10597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.912417 10597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9abeba26db814eddb59f9fe474aa1317, State: Initialized, Role: FOLLOWER
I20260926 07:02:16.912578 10597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.912650 10597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:16.912700 10597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:16.912747 10597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:02:16.913717 10597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.913837 10597 leader_election.cc:302] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9abeba26db814eddb59f9fe474aa1317; no voters:
I20260926 07:02:16.914029 10597 leader_election.cc:288] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:02:16.914065 10601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:02:16.914171 10601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [term 2 LEADER]: Becoming Leader. State: Replica: 9abeba26db814eddb59f9fe474aa1317, State: Running, Role: LEADER
I20260926 07:02:16.914253 10601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } }
I20260926 07:02:16.914301 10597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:16.914507 10601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9abeba26db814eddb59f9fe474aa1317. Latest consensus state: current_term: 2 leader_uuid: "9abeba26db814eddb59f9fe474aa1317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } } }
I20260926 07:02:16.914572 10601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:16.914525 10602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9abeba26db814eddb59f9fe474aa1317" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9abeba26db814eddb59f9fe474aa1317" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42403 } } }
I20260926 07:02:16.914731 10602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:16.914760 10605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:16.915347 10605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:16.915512 10605 catalog_manager.cc:1259] Loaded cluster ID: b3db420885e8445e8c74c81f5625d456
I20260926 07:02:16.915750 10605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:16.916975 10605 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:16.917150 10605 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9abeba26db814eddb59f9fe474aa1317: Loaded TSK: 0
I20260926 07:02:16.917359 10605 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:02:16.982574 10599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:16.982831 10599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:16.982873 10599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:16.984067 10599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:16.984133 10599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:16.984179 10599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:16.985688 10599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.65.keytab
--principal=oryx/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:43321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=33487
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:16.985946 10599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:16.986196 10599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:16.991945 10624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.991946 10623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:16.991971 10626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:16.992364 10599 server_base.cc:1034] running on GCE node
I20260926 07:02:16.992565 10599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:16.992859 10599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:16.994050 10599 hybrid_clock.cc:648] HybridClock initialized: now 1790406136994023 us; error 34 us; skew 500 ppm
Sep 26 07:02:16 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406136, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:16.996696 10599 init.cc:377] Logged in from keytab as oryx/127.1.217.65@KRBTEST.COM (short username oryx)
I20260926 07:02:16.997259 10599 webserver.cc:460] Webserver started at http://127.1.217.65:33487/ using document root <none> and password file <none>
I20260926 07:02:16.997510 10599 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:16.997594 10599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:16.999329 10599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:16.999979 10633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.000162 10599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:17.000268 10599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "08a0a89a53a64c80b663d2aef66d726a"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "725c1e3b5d077eb5725cf0fb114ea5be"
server_key_iv: "856d1be3c22e81177a9210d8fcca9688"
server_key_version: "encryptionkey@0"
I20260926 07:02:17.000612 10599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:17.011404 10599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:17.013641 10599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:17.013813 10599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:17.014142 10599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:17.014568 10599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:17.014602 10599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.014642 10599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:17.014668 10599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.020951 10599 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43321
I20260926 07:02:17.020992 10746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43321 every 8 connection(s)
I20260926 07:02:17.021330 10599 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:17.026609 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10599
I20260926 07:02:17.026921 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:38195
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:42403
--webserver_port=35937
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.030791 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.021973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:38147)
0926 07:02:17.022189 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.022237 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.022833 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.023467 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.023470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.024046 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.024237 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.024241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.025883 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.025883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.026660 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.026662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.027437 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.027438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.027572 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.028119 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.028127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.030559 (+ 2432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":109,"threads_started":1}
W20260926 07:02:17.031057 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.022108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:38147 (local address 127.1.217.126:42403)
0926 07:02:17.023042 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.023051 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.023088 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:02:17.023221 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.023228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.023282 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.023423 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.024374 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.025786 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.026797 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.027341 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.027463 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.030715 (+ 3252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:38147: BlockingRecv error: recv got EOF from 127.1.217.65:38147 (error 108)
Metrics: {"server-negotiator.queue_time_us":881,"thread_start_us":70,"threads_started":1}
W20260926 07:02:17.031104 10747 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:42403 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.217.126:42403: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.036952 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.031342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:55441)
0926 07:02:17.031400 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.031411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.031457 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.031632 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.031633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.031796 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.031853 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.031854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.032728 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.032729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.033674 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.033677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.034163 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.034164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.034254 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.034715 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.034720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.036861 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:02:17.037042 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.031421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55441 (local address 127.1.217.126:42403)
0926 07:02:17.031523 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.031525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.031531 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:02:17.031550 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.031551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.031561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.031610 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.031935 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.032638 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.033782 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.034068 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.034139 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.036967 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:55441: BlockingRecv error: recv got EOF from 127.1.217.65:55441 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.042768 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.037627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:47701)
0926 07:02:17.037723 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.037733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.037785 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.037920 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.037920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.038055 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.038086 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.038087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.039042 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.039043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.039828 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.039831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.040302 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.040304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.040353 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.040837 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.040842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.042698 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 07:02:17.042850 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.037684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:47701 (local address 127.1.217.126:42403)
0926 07:02:17.037729 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.037731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.037772 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:02:17.037835 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.037836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.037848 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.037916 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.038168 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.038955 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.039916 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.040224 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.040287 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.042790 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:47701: BlockingRecv error: recv got EOF from 127.1.217.65:47701 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 07:02:17.043129 10747 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:02:17.113088 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:17.113310 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:17.113348 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:17.114586 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:17.114647 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:17.114699 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:17.116149 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.66.keytab
--principal=oryx/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:38195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=35937
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:17.116398 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:17.116612 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:17.121505 10757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:17.121518 10759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:17.121511 10756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:17.121876 10751 server_base.cc:1034] running on GCE node
I20260926 07:02:17.122103 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:17.122359 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:17.123517 10751 hybrid_clock.cc:648] HybridClock initialized: now 1790406137123503 us; error 24 us; skew 500 ppm
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406137, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:17.126112 10751 init.cc:377] Logged in from keytab as oryx/127.1.217.66@KRBTEST.COM (short username oryx)
I20260926 07:02:17.126562 10751 webserver.cc:460] Webserver started at http://127.1.217.66:35937/ using document root <none> and password file <none>
I20260926 07:02:17.126748 10751 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:17.126799 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:17.128057 10751 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:17.128616 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.128847 10751 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:02:17.128944 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "830602a0468b4cfdaea8ea074d0fde8b"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "33c69882cd628f4b05e23354691fd795"
server_key_iv: "ce75022df480620a41c3f39b36a32f0b"
server_key_version: "encryptionkey@0"
I20260926 07:02:17.129297 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:17.135838 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:17.137607 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:17.137756 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:17.137960 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:17.138247 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:17.138279 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.138300 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:17.138345 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.143895 10751 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:38195
I20260926 07:02:17.143957 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:38195 every 8 connection(s)
I20260926 07:02:17.144284 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.151140 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.144968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:36097)
0926 07:02:17.145120 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.145159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.145618 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.145822 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.145825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.146364 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.146545 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.146550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.147159 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.147160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.147927 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.147929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.148318 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.148319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.148414 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.148956 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.148964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.151007 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260926 07:02:17.151141 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.144998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:36097 (local address 127.1.217.126:42403)
0926 07:02:17.145039 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.145041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.145241 (+ 200us) server_negotiation.cc:407] Connection header received
0926 07:02:17.145696 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.145697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.145709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.145761 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.146617 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.147085 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.148006 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.148242 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.148309 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.151096 (+ 2787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:36097: BlockingRecv error: recv got EOF from 127.1.217.66:36097 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:02:17.151347 10880 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:42403 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.217.126:42403: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
I20260926 07:02:17.153247 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10751
I20260926 07:02:17.153571 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:41113
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:42403
--webserver_port=45093
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.155833 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.151546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:54821)
0926 07:02:17.151591 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.151601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.151641 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.151765 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.151766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.151865 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.151902 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.151903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.152458 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.152459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.152992 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.152994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.153486 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.153487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.153544 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.154000 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.154004 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.155765 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:02:17.156025 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.151607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:54821 (local address 127.1.217.126:42403)
0926 07:02:17.151662 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.151664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.151672 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:17.151698 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.151699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.151711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.151761 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.151956 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.152404 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.153098 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.153398 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.153490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.155928 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:54821: BlockingRecv error: recv got EOF from 127.1.217.66:54821 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.160737 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.156119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:33951)
0926 07:02:17.156237 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.156247 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.156297 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.156416 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.156416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.156512 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.156534 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.156536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.157472 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.157473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.157964 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.157965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.158456 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.158457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.158495 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.158930 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.158934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.160634 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260926 07:02:17.160805 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.156199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:33951 (local address 127.1.217.126:42403)
0926 07:02:17.156261 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.156263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.156288 (+ 25us) server_negotiation.cc:407] Connection header received
0926 07:02:17.156337 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.156338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.156348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.156414 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.156597 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.157388 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.158045 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.158368 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.158439 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.160723 (+ 2284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:33951: BlockingRecv error: recv got EOF from 127.1.217.66:33951 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 07:02:17.160909 10880 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:02:17.235342 10883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:17.235566 10883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:17.235602 10883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:17.236732 10883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:17.236827 10883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:17.236886 10883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:17.238301 10883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/krb5kdc/oryx_127.1.217.67.keytab
--principal=oryx/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:41113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=45093
--tserver_master_addrs=127.1.217.126:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:17.238586 10883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:17.238842 10883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:17.243628 10888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:17.243698 10889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:17.243860 10883 server_base.cc:1034] running on GCE node
W20260926 07:02:17.243974 10891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:17.244186 10883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:17.244370 10883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:17.245538 10883 hybrid_clock.cc:648] HybridClock initialized: now 1790406137245529 us; error 41 us; skew 500 ppm
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406137, etypes {rep=17 tkt=17 ses=17}, oryx/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:17.247931 10883 init.cc:377] Logged in from keytab as oryx/127.1.217.67@KRBTEST.COM (short username oryx)
I20260926 07:02:17.248374 10883 webserver.cc:460] Webserver started at http://127.1.217.67:45093/ using document root <none> and password file <none>
I20260926 07:02:17.248602 10883 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:17.248679 10883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:17.249989 10883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:17.250527 10898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.250694 10883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:17.250787 10883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "98e24fa85af74417b8915ecda356d6fd"
format_stamp: "Formatted at 2026-09-26 07:02:16 on dist-test-slave-t1mc"
server_key: "b36d01953bc7b85fd9cb5fe35ee7cb68"
server_key_iv: "2ae3db3eed5a33472f38d573089af6e6"
server_key_version: "encryptionkey@0"
I20260926 07:02:17.251120 10883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:17.270833 10883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:17.272655 10883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:17.272869 10883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:17.273092 10883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:17.273391 10883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:17.273443 10883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.273486 10883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:17.273514 10883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:17.279474 10883 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:41113
I20260926 07:02:17.279520 11011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:41113 every 8 connection(s)
I20260926 07:02:17.279855 10883 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
W20260926 07:02:17.286942 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.280526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:56961 (local address 127.1.217.126:42403)
0926 07:02:17.280586 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.280588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.280817 (+ 229us) server_negotiation.cc:407] Connection header received
0926 07:02:17.281347 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.281348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.281361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.281431 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.282392 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.282881 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.283883 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.284080 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.284165 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.286897 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:56961: BlockingRecv error: recv got EOF from 127.1.217.67:56961 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:02:17.286954 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.280520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:56961)
0926 07:02:17.280668 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.280712 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.281240 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.281512 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.281515 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.282081 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.282272 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.282276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.282976 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.282977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.283788 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.283790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.284173 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.284174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.284298 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.284820 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.284828 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.286819 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
W20260926 07:02:17.287266 11012 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:42403 (0 consecutive failures): Not authorized: Failed to ping master at 127.1.217.126:42403: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
I20260926 07:02:17.289047 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 10883
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.292042 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.287512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:42411)
0926 07:02:17.287564 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.287574 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.287616 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.287814 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.287815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.287942 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.287981 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.287982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.288596 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.288597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.289231 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.289234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.289700 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.289701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.289780 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.290205 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.290209 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.291961 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 07:02:17.292140 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.287653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:42411 (local address 127.1.217.126:42403)
0926 07:02:17.287715 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.287716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.287722 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:02:17.287739 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.287740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.287747 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.287809 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.288065 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.288519 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.289324 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.289611 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.289689 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.292059 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:42411: BlockingRecv error: recv got EOF from 127.1.217.67:42411 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:17 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:17.296602 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.292259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:60281)
0926 07:02:17.292292 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:17.292300 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:17.292330 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:17.292468 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:17.292468 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:17.292587 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:17.292609 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.292610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.293501 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.293502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.294149 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:17.294151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.294525 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.294526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:17.294585 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.294884 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:17.294889 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:17.296545 (+ 1656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 07:02:17.296700 11012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:02:17.296702 10750 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:17.292324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:60281 (local address 127.1.217.126:42403)
0926 07:02:17.292354 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.292356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.292364 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:17.292383 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.292384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.292391 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.292450 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.292683 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.293421 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.294249 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.294453 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.294503 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.296621 (+ 2118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:60281: BlockingRecv error: recv got EOF from 127.1.217.67:60281 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:02:17.297365 11015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:17.289426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.1.217.126:42403)
0926 07:02:17.289547 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:17.289549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:17.289640 (+ 91us) server_negotiation.cc:407] Connection header received
0926 07:02:17.289701 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:17.289701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:17.289712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:17.289784 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:17.290342 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.290967 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.292023 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:17.292391 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:17.292455 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:17.293765 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:17.293769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:17.293770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:17.293780 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:17.295627 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:17.296203 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:17.296206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:17.296206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:17.296323 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:17.296521 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:17.296522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:17.296522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:17.296929 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:17.297032 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:17.297147 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:02:17.297231 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:18.047739 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:18.043505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:58693)
0926 07:02:18.043645 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:18.043654 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:18.043700 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:18.043814 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:18.043815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:18.043962 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:18.044008 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.044009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.044647 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.044648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.045191 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.045193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.045517 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.045517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.045556 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.045970 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:18.045974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:18.047673 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
W20260926 07:02:18.047886 11018 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:18.043572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:58693 (local address 127.1.217.126:42403)
0926 07:02:18.043695 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:18.043697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:18.043708 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:02:18.043736 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:18.043737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:18.043746 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:18.043809 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:18.044068 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.044573 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.045265 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.045453 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.045511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.047784 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:58693: BlockingRecv error: recv got EOF from 127.1.217.65:58693 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:18.165946 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:18.161283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:55841)
0926 07:02:18.161465 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:18.161478 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:18.161548 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:18.161707 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:18.161707 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:18.161883 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:18.161940 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.161941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.162597 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.162598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.163164 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.163166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.163563 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.163565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.163632 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.164098 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:18.164103 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:18.165844 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
W20260926 07:02:18.165987 11018 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:18.161426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:55841 (local address 127.1.217.126:42403)
0926 07:02:18.161476 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:18.161479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:18.161528 (+ 49us) server_negotiation.cc:407] Connection header received
0926 07:02:18.161603 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:18.161604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:18.161617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:18.161690 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:18.162019 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.162497 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.163252 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.163444 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.163556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.165924 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:55841: BlockingRecv error: recv got EOF from 127.1.217.66:55841 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:18 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:18.301625 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:18.297081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:33335)
0926 07:02:18.297279 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:18.297291 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:18.297347 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:18.297474 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:18.297475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:18.297642 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:18.297690 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.297691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.298328 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.298329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.298860 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:18.298862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.299182 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.299183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:18.299220 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.299624 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:18.299629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:18.301548 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
W20260926 07:02:18.301720 11018 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:18.297178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:33335 (local address 127.1.217.126:42403)
0926 07:02:18.297249 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:18.297250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:18.297351 (+ 101us) server_negotiation.cc:407] Connection header received
0926 07:02:18.297405 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:18.297406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:18.297417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:18.297462 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:18.297750 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.298251 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.298930 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:18.299123 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:18.299172 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:18.301632 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:33335: BlockingRecv error: recv got EOF from 127.1.217.67:33335 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:19.059455 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:19.048291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:54727)
0926 07:02:19.048436 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:19.048448 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:19.048530 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:19.048703 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:19.048704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:19.048919 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:19.048988 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.048991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.049642 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.049643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.050179 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.050180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.050593 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.050595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.050654 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.057527 (+ 6873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:19.057532 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:19.059361 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260926 07:02:19.059564 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:19.048357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:54727 (local address 127.1.217.126:42403)
0926 07:02:19.048535 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:19.048537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:19.048546 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:19.048601 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:19.048601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:19.048615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:19.048688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:19.049068 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.049569 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.050260 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.050486 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.050544 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.059482 (+ 8938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:54727: BlockingRecv error: recv got EOF from 127.1.217.65:54727 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:19.171177 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:19.166479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:52755)
0926 07:02:19.166655 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:19.166664 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:19.166722 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:19.166860 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:19.166860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:19.167035 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:19.167110 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.167112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.167969 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.167970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.168493 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.168495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.168854 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.168855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.168899 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.169317 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:19.169321 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:19.171119 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260926 07:02:19.171305 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:19.166556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:52755 (local address 127.1.217.126:42403)
0926 07:02:19.166620 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:19.166622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:19.166723 (+ 101us) server_negotiation.cc:407] Connection header received
0926 07:02:19.166786 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:19.166787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:19.166798 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:19.166841 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:19.167197 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.167900 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.168567 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.168744 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.168817 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.171225 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:52755: BlockingRecv error: recv got EOF from 127.1.217.66:52755 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:19 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:19.306805 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:19.302090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:33619)
0926 07:02:19.302217 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:19.302229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:19.302290 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:19.302463 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:19.302464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:19.302639 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:19.302686 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.302687 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.303471 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.303472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.304005 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:19.304007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.304416 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.304417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:19.304479 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.304955 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:19.304959 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:19.306710 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
W20260926 07:02:19.306846 11022 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:19.302209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:33619 (local address 127.1.217.126:42403)
0926 07:02:19.302258 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:19.302260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:19.302272 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:02:19.302359 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:19.302360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:19.302373 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:19.302442 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:19.302761 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.303352 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.304091 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:19.304336 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:19.304412 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:19.306784 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:33619: BlockingRecv error: recv got EOF from 127.1.217.67:33619 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:20.064555 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:20.060000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:47983)
0926 07:02:20.060129 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:20.060144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:20.060195 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:20.060361 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:20.060362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:20.060530 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:20.060577 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.060579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.061299 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.061300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.061848 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.061849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.062199 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.062199 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.062239 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.062700 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:20.062704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:20.064456 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
W20260926 07:02:20.064625 11026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:20.060067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:47983 (local address 127.1.217.126:42403)
0926 07:02:20.060200 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:20.060202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:20.060214 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:02:20.060245 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:20.060245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:20.060255 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:20.060323 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:20.060657 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.061199 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.061934 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.062132 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.062188 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.064543 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:47983: BlockingRecv error: recv got EOF from 127.1.217.65:47983 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:20.176614 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:20.171636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:57313)
0926 07:02:20.171828 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:20.171840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:20.171933 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:20.172098 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:20.172099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:20.172273 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:20.172316 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.172318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.173086 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.173087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.173617 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.173619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.174034 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.174035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.174102 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.174552 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:20.174556 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:20.176555 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
W20260926 07:02:20.176844 11026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:20.171720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:57313 (local address 127.1.217.126:42403)
0926 07:02:20.171814 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:20.171817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:20.171913 (+ 96us) server_negotiation.cc:407] Connection header received
0926 07:02:20.172007 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:20.172008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:20.172026 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:20.172088 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:20.172428 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.173004 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.173723 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.173928 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.174034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.176719 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:57313: BlockingRecv error: recv got EOF from 127.1.217.66:57313 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:20 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:20.311724 11028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:20.307284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:34091)
0926 07:02:20.307464 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:20.307479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:20.307546 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:20.307681 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:20.307682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:20.307849 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:20.307895 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.307896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.308528 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.308529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.309104 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:20.309106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.309422 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.309422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:20.309460 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.309871 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:20.309876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:20.311668 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":127,"threads_started":1}
W20260926 07:02:20.311860 11026 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:20.307392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:34091 (local address 127.1.217.126:42403)
0926 07:02:20.307439 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:20.307442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:20.307580 (+ 138us) server_negotiation.cc:407] Connection header received
0926 07:02:20.307601 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:20.307602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:20.307613 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:20.307655 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:20.307963 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.308461 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.309173 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:20.309361 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:20.309410 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:20.311780 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:34091: BlockingRecv error: recv got EOF from 127.1.217.67:34091 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:21.069952 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:21.065026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:44389)
0926 07:02:21.065187 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:21.065197 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:21.065254 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:21.065597 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:21.065599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:21.065759 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:21.065808 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.065810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.066466 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.066466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.066980 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.066981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.067361 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.067362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.067400 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.067863 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:21.067868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:21.069893 (+ 2025us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
W20260926 07:02:21.070086 11030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:21.065107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:44389 (local address 127.1.217.126:42403)
0926 07:02:21.065212 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:21.065214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:21.065238 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:02:21.065486 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:21.065488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:21.065499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:21.065569 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:21.065890 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.066398 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.067072 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.067273 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.067329 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.069977 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:44389: BlockingRecv error: recv got EOF from 127.1.217.65:44389 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:21.181316 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:21.177048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:47087)
0926 07:02:21.177169 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:21.177187 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:21.177239 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:21.177389 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:21.177390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:21.177543 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:21.177589 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.177590 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.178208 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.178209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.178744 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.178746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.179069 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.179070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.179106 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.179543 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:21.179547 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:21.181257 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260926 07:02:21.181456 11030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:21.177118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:47087 (local address 127.1.217.126:42403)
0926 07:02:21.177160 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:21.177161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:21.177225 (+ 64us) server_negotiation.cc:407] Connection header received
0926 07:02:21.177309 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:21.177310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:21.177320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:21.177370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:21.177656 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.178144 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.178818 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.179003 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.179059 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.181373 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:47087: BlockingRecv error: recv got EOF from 127.1.217.66:47087 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:21 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:21.317206 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:21.312186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:42035)
0926 07:02:21.312334 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:21.312344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:21.312392 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:21.312584 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:21.312585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:21.312748 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:21.312870 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.312872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.313610 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.313611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.314189 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:21.314191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.314652 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.314653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:21.314696 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.315151 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:21.315155 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:21.317149 (+ 1994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260926 07:02:21.317335 11030 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:21.312319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:42035 (local address 127.1.217.126:42403)
0926 07:02:21.312368 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:21.312375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:21.312385 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:21.312438 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:21.312439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:21.312452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:21.312514 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:21.312953 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.313520 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.314327 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:21.314562 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:21.314658 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:21.317236 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:42035: BlockingRecv error: recv got EOF from 127.1.217.67:42035 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:22.074976 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:22.070390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:32781)
0926 07:02:22.070543 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:22.070554 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:22.070623 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:22.070782 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:22.070783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:22.070938 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:22.070986 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.070988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.071610 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.071611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.072124 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.072125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.072517 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.072518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.072564 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.073063 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:22.073068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:22.074919 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
W20260926 07:02:22.075086 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:22.070469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:32781 (local address 127.1.217.126:42403)
0926 07:02:22.070565 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:22.070567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:22.070606 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:02:22.070684 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:22.070684 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:22.070695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:22.070744 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:22.071064 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.071542 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.072220 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.072420 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.072482 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.075008 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:32781: BlockingRecv error: recv got EOF from 127.1.217.65:32781 (error 108)
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":44,"threads_started":1}
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:22.186496 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:22.181831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:34613)
0926 07:02:22.181978 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:22.181992 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:22.182059 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:22.182219 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:22.182220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:22.182394 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:22.182449 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.182469 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.183207 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.183208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.183743 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.183745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.184134 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.184135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.184193 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.184723 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:22.184728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:22.186435 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
W20260926 07:02:22.186587 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:22.181963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:34613 (local address 127.1.217.126:42403)
0926 07:02:22.182044 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:22.182046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:22.182056 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:02:22.182107 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:22.182107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:22.182118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:22.182180 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:22.182575 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.183076 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.183839 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.184034 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.184121 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.186524 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:34613: BlockingRecv error: recv got EOF from 127.1.217.66:34613 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:22 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:22.321987 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:22.317620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:44729)
0926 07:02:22.317787 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:22.317797 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:22.317848 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:22.317971 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:22.317972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:22.318118 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:22.318167 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.318169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.318782 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.318783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.319304 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:22.319306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.319625 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.319626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:22.319670 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.320188 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:22.320192 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:22.321930 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260926 07:02:22.322136 11034 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:22.317715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:44729 (local address 127.1.217.126:42403)
0926 07:02:22.317792 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:22.317793 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:22.317849 (+ 56us) server_negotiation.cc:407] Connection header received
0926 07:02:22.317902 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:22.317903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:22.317914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:22.317952 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:22.318260 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.318712 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.319385 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:22.319562 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:22.319616 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:22.322049 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:44729: BlockingRecv error: recv got EOF from 127.1.217.67:44729 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:23.080124 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:23.075476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:57011)
0926 07:02:23.075611 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:23.075621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:23.075671 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:23.075838 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:23.075839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:23.075994 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:23.076044 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.076045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.076692 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.076693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.077250 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.077251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.077619 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.077620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.077673 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.078251 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:23.078256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:23.080029 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
W20260926 07:02:23.080174 11038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:23.075519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:57011 (local address 127.1.217.126:42403)
0926 07:02:23.075655 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:23.075657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:23.075669 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:02:23.075740 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:23.075741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:23.075751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:23.075826 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:23.076142 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.076630 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.077337 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.077522 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.077574 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.080104 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:57011: BlockingRecv error: recv got EOF from 127.1.217.65:57011 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:23.191288 11039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:23.187082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:50815)
0926 07:02:23.187232 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:23.187242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:23.187303 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:23.187422 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:23.187423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:23.187575 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:23.187621 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.187622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.188220 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.188221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.188761 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.188785 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.189114 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.189115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.189154 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.189588 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:23.189592 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:23.191231 (+ 1639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
W20260926 07:02:23.191418 11038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:23.187101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50815 (local address 127.1.217.126:42403)
0926 07:02:23.187162 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:23.187164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:23.187298 (+ 134us) server_negotiation.cc:407] Connection header received
0926 07:02:23.187348 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:23.187348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:23.187359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:23.187408 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:23.187674 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.188158 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.188843 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.189048 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.189107 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.191336 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:50815: BlockingRecv error: recv got EOF from 127.1.217.66:50815 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:23 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:23.327049 11040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:23.322444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:43017)
0926 07:02:23.322628 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:23.322641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:23.322737 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:23.322899 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:23.322900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:23.323098 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:23.323148 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.323150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.323817 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.323819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.324401 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:23.324402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.324759 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.324759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:23.324851 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.325323 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:23.325328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:23.326991 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":97,"threads_started":1}
W20260926 07:02:23.327193 11038 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:23.322537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:43017 (local address 127.1.217.126:42403)
0926 07:02:23.322617 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:23.322620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:23.322719 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:02:23.322809 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:23.322810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:23.322825 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:23.322893 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:23.323248 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.323739 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.324487 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:23.324679 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:23.324758 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:23.327081 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:43017: BlockingRecv error: recv got EOF from 127.1.217.67:43017 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:24.093755 11041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:24.080629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:49787)
0926 07:02:24.080818 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:24.080832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:24.080909 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:24.081079 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:24.081080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:24.081237 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:24.081285 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.081286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.081953 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.081954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.082500 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.082502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.082834 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.082835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.082874 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.091738 (+ 8864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:24.091743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:24.093658 (+ 1915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
W20260926 07:02:24.093827 11042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:24.080682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:49787 (local address 127.1.217.126:42403)
0926 07:02:24.080859 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:24.080861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:24.080885 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:02:24.080962 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:24.080963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:24.080974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:24.081064 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:24.081368 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.081877 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.082582 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.082770 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.082820 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.093750 (+ 10930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:49787: BlockingRecv error: recv got EOF from 127.1.217.65:49787 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:24.197031 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:24.191719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:56347)
0926 07:02:24.191912 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:24.191925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:24.192029 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:24.192157 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:24.192158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:24.192342 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:24.192388 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.192389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.193104 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.193105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.193744 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.193748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.194094 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.194095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.194152 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.194598 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:24.194605 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:24.196913 (+ 2308us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260926 07:02:24.197103 11042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:24.191811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:56347 (local address 127.1.217.126:42403)
0926 07:02:24.191903 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:24.191906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:24.192007 (+ 101us) server_negotiation.cc:407] Connection header received
0926 07:02:24.192085 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:24.192086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:24.192099 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:24.192153 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:24.192465 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.193028 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.193810 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.194004 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.194085 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.197036 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:56347: BlockingRecv error: recv got EOF from 127.1.217.66:56347 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:24 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:24.332227 11044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:24.327517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:36413)
0926 07:02:24.327682 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:24.327695 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:24.327778 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:24.327943 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:24.327944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:24.328118 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:24.328164 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.328166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.328858 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.328859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.329435 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:24.329436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.329801 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.329802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:24.329865 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.330289 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:24.330293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:24.332158 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
W20260926 07:02:24.332340 11042 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:24.327598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:36413 (local address 127.1.217.126:42403)
0926 07:02:24.327673 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:24.327675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:24.327759 (+ 84us) server_negotiation.cc:407] Connection header received
0926 07:02:24.327867 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:24.327868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:24.327881 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:24.327931 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:24.328241 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.328746 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.329515 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:24.329721 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:24.329801 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:24.332246 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:36413: BlockingRecv error: recv got EOF from 127.1.217.67:36413 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:25.098888 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:25.094279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:50969)
0926 07:02:25.094429 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:25.094440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:25.094490 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:25.094673 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:25.094673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:25.094837 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:25.094884 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.094886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.095510 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.095511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.096069 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.096071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.096476 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.096477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.096528 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.097030 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:25.097035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:25.098814 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260926 07:02:25.099076 11046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:25.094405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:50969 (local address 127.1.217.126:42403)
0926 07:02:25.094546 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:25.094548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:25.094560 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:02:25.094582 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:25.094582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:25.094592 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:25.094659 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:25.094950 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.095445 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.096169 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.096367 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.096437 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.098992 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:50969: BlockingRecv error: recv got EOF from 127.1.217.65:50969 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:25.202379 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:25.197517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:59099)
0926 07:02:25.197664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:25.197675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:25.197738 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:25.197869 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:25.197869 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:25.198025 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:25.198077 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.198079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.199139 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.199140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.199654 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.199655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.199974 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.199975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.200014 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.200474 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:25.200479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:25.202320 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
W20260926 07:02:25.202530 11046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:25.197571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:59099 (local address 127.1.217.126:42403)
0926 07:02:25.197643 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:25.197645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:25.197738 (+ 93us) server_negotiation.cc:407] Connection header received
0926 07:02:25.197784 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:25.197784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:25.197796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:25.197856 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:25.198140 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.199074 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.199723 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.199913 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.199967 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.202454 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:59099: BlockingRecv error: recv got EOF from 127.1.217.66:59099 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:25 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:25.337292 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:25.332673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:56129)
0926 07:02:25.332899 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:25.332911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:25.332978 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:25.333096 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:25.333097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:25.333250 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:25.333297 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.333299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.333909 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.333910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.334427 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:25.334429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.334741 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.334742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:25.334778 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.335236 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:25.335241 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:25.337236 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":128,"threads_started":1}
W20260926 07:02:25.337460 11046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:25.332724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:56129 (local address 127.1.217.126:42403)
0926 07:02:25.332800 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:25.332803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:25.332973 (+ 170us) server_negotiation.cc:407] Connection header received
0926 07:02:25.333020 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:25.333021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:25.333032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:25.333081 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:25.333351 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.333847 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.334492 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:25.334672 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:25.334725 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:25.337343 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:56129: BlockingRecv error: recv got EOF from 127.1.217.67:56129 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:26.103850 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:26.099318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:52591)
0926 07:02:26.099472 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:26.099485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:26.099560 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:26.099692 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:26.099693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:26.099851 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:26.099898 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.099900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.100509 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.100509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.101065 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.101067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.101406 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.101406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.101443 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.101976 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:26.101981 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:26.103792 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
W20260926 07:02:26.104012 11050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:26.099392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:52591 (local address 127.1.217.126:42403)
0926 07:02:26.099519 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:26.099521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:26.099530 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:26.099613 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:26.099613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:26.099623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:26.099673 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:26.099958 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.100447 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.101140 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.101342 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.101399 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.103906 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:52591: BlockingRecv error: recv got EOF from 127.1.217.65:52591 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:26.207513 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:26.202873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:45579)
0926 07:02:26.202999 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:26.203009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:26.203071 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:26.203203 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:26.203204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:26.203352 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:26.203414 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.203416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.204030 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.204030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.204834 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.204837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.205144 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.205145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.205181 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.205592 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:26.205596 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:26.207393 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260926 07:02:26.207541 11050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:26.202911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:45579 (local address 127.1.217.126:42403)
0926 07:02:26.202983 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:26.202985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:26.203066 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:02:26.203128 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:26.203129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:26.203140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:26.203184 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:26.203476 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.203964 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.204888 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.205086 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.205140 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.207480 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:45579: BlockingRecv error: recv got EOF from 127.1.217.66:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:26 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:26.342236 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:26.337788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:48031)
0926 07:02:26.337954 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:26.337964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:26.338051 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:26.338203 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:26.338204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:26.338380 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:26.338426 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.338427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.339078 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.339079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.339602 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:26.339604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.339949 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.339950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:26.340007 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.340452 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:26.340456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:26.342170 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260926 07:02:26.342491 11050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:26.337859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:48031 (local address 127.1.217.126:42403)
0926 07:02:26.337926 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:26.337928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:26.338032 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:02:26.338105 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:26.338106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:26.338120 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:26.338180 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:26.338500 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.338989 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.339682 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:26.339868 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:26.339945 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:26.342433 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:48031: BlockingRecv error: recv got EOF from 127.1.217.67:48031 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:27.108731 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:27.104363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:58453)
0926 07:02:27.104479 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:27.104491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:27.104565 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:27.104696 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:27.104696 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:27.104876 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:27.104924 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.104925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.105564 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.105564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.106079 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.106081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.106404 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.106405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.106443 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.106904 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:27.106909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:27.108666 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
W20260926 07:02:27.108882 11054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:27.104434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:58453 (local address 127.1.217.126:42403)
0926 07:02:27.104574 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:27.104576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:27.104584 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:27.104606 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:27.104607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:27.104617 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:27.104680 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:27.104988 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.105483 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.106151 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.106345 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.106396 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.108759 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:58453: BlockingRecv error: recv got EOF from 127.1.217.65:58453 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":64,"threads_started":1}
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:27.212461 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:27.208093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:42127)
0926 07:02:27.208227 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:27.208237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:27.208295 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:27.208413 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:27.208414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:27.208580 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:27.208626 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.208627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.209317 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.209318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.209832 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.209834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.210147 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.210148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.210184 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.210614 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:27.210619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:27.212403 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
W20260926 07:02:27.212553 11054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:27.208155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:42127 (local address 127.1.217.126:42403)
0926 07:02:27.208217 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:27.208220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:27.208301 (+ 81us) server_negotiation.cc:407] Connection header received
0926 07:02:27.208332 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:27.208333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:27.208343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:27.208400 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:27.208686 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.209255 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.209902 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.210088 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.210141 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.212491 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:42127: BlockingRecv error: recv got EOF from 127.1.217.66:42127 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:27 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:27.347321 11056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:27.342753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:51669)
0926 07:02:27.342895 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:27.342905 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:27.342963 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:27.343109 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:27.343110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:27.343281 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:27.343330 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.343332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.343992 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.343993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.344505 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:27.344507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.344896 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.344897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:27.344946 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.345440 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:27.345451 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:27.347235 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
W20260926 07:02:27.347404 11054 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:27.342924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:51669 (local address 127.1.217.126:42403)
0926 07:02:27.342988 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:27.342990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:27.343001 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:02:27.343030 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:27.343031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:27.343042 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:27.343093 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:27.343405 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.343924 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.344579 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:27.344802 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:27.344896 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:27.347343 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:51669: BlockingRecv error: recv got EOF from 127.1.217.67:51669 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:28.114148 11057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:28.109224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:51415)
0926 07:02:28.109407 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:28.109417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:28.109468 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:28.109608 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:28.109608 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:28.109798 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:28.109844 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.109845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.110461 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.110461 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.111011 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.111014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.111351 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.111352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.111389 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.111921 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:28.111925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:28.114067 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
W20260926 07:02:28.114212 11058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:28.109341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:51415 (local address 127.1.217.126:42403)
0926 07:02:28.109474 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:28.109476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:28.109511 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:02:28.109531 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:28.109532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:28.109542 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:28.109593 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:28.109905 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.110400 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.111096 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.111291 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.111344 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.114158 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:51415: BlockingRecv error: recv got EOF from 127.1.217.65:51415 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:28.217872 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:28.212968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:48547)
0926 07:02:28.213102 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:28.213112 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:28.213199 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:28.213341 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:28.213342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:28.213520 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:28.213561 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.213563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.214349 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.214350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.215015 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.215019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.215447 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.215448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.215511 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.215961 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:28.215984 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:28.217765 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
W20260926 07:02:28.217902 11058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:28.213023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:48547 (local address 127.1.217.126:42403)
0926 07:02:28.213072 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:28.213074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:28.213178 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:02:28.213264 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:28.213264 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:28.213276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:28.213334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:28.213645 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.214275 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.215091 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.215365 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.215446 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.217839 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:48547: BlockingRecv error: recv got EOF from 127.1.217.66:48547 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:28 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:28.352747 11060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:28.347969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:38871)
0926 07:02:28.348239 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:28.348252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:28.348349 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:28.348519 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:28.348520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:28.348741 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:28.348817 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.348819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.349481 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.349482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.350018 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:28.350020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.350362 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.350362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:28.350399 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.350866 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:28.350876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:28.352624 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
W20260926 07:02:28.352831 11058 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:28.348024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:38871 (local address 127.1.217.126:42403)
0926 07:02:28.348085 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:28.348088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:28.348353 (+ 265us) server_negotiation.cc:407] Connection header received
0926 07:02:28.348428 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:28.348430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:28.348444 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:28.348503 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:28.348890 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.349391 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.350098 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:28.350295 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:28.350357 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:28.352735 (+ 2378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:38871: BlockingRecv error: recv got EOF from 127.1.217.67:38871 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:29.124330 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:29.114610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:40995)
0926 07:02:29.114769 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:29.114781 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:29.114858 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:29.114999 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:29.114999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:29.115265 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:29.115310 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.115312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.115952 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.115953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.116474 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.116476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.116852 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.116853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.116899 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.122384 (+ 5485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:29.122389 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:29.124272 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260926 07:02:29.124469 11062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:29.114692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:40995 (local address 127.1.217.126:42403)
0926 07:02:29.114810 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:29.114812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:29.114852 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:02:29.114915 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:29.114916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:29.114925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:29.114987 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:29.115373 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.115871 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.116551 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.116754 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.116827 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.124377 (+ 7550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:40995: BlockingRecv error: recv got EOF from 127.1.217.65:40995 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":47,"threads_started":1}
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:29.222925 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:29.218299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:32775)
0926 07:02:29.218435 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:29.218445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:29.218502 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:29.218628 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:29.218629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:29.218805 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:29.218850 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.218851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.219646 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.219647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.220195 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.220197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.220498 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.220498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.220536 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.221072 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:29.221076 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:29.222817 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
W20260926 07:02:29.222952 11062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:29.218372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:32775 (local address 127.1.217.126:42403)
0926 07:02:29.218433 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:29.218435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:29.218499 (+ 64us) server_negotiation.cc:407] Connection header received
0926 07:02:29.218559 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:29.218560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:29.218570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:29.218611 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:29.218918 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.219580 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.220261 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.220440 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.220488 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.222891 (+ 2403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:32775: BlockingRecv error: recv got EOF from 127.1.217.66:32775 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:29 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:29.357736 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:29.353295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:44005)
0926 07:02:29.353539 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:29.353549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:29.353610 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:29.353729 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:29.353733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:29.353906 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:29.353951 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.353952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.354556 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.354557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.355085 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:29.355086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.355396 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.355397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:29.355433 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.355879 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:29.355883 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:29.357630 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":193,"threads_started":1}
W20260926 07:02:29.357782 11062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:29.353378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:44005 (local address 127.1.217.126:42403)
0926 07:02:29.353435 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:29.353437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:29.353623 (+ 186us) server_negotiation.cc:407] Connection header received
0926 07:02:29.353652 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:29.353653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:29.353667 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:29.353716 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:29.354021 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.354501 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.355155 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:29.355330 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:29.355391 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:29.357717 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:44005: BlockingRecv error: recv got EOF from 127.1.217.67:44005 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:30.128980 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:30.124745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:44947)
0926 07:02:30.124909 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:30.124919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:30.124969 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:30.125112 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:30.125112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:30.125280 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:30.125323 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.125325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.125988 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.125988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.126507 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.126508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.126813 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.126814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.126852 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.127249 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:30.127253 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:30.128911 (+ 1658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
W20260926 07:02:30.129068 11066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:30.124805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:44947 (local address 127.1.217.126:42403)
0926 07:02:30.124937 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:30.124939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:30.124953 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:02:30.125025 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:30.125026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:30.125039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:30.125098 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:30.125408 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.125925 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.126572 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.126756 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.126805 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.128999 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:44947: BlockingRecv error: recv got EOF from 127.1.217.65:44947 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:30.227550 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:30.223363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:54439)
0926 07:02:30.223527 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:30.223536 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:30.223587 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:30.223709 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:30.223709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:30.223866 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:30.223912 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.223914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.224520 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.224521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.225089 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.225091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.225393 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.225394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.225430 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.225855 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:30.225859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:30.227492 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
W20260926 07:02:30.227682 11066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:30.223455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:54439 (local address 127.1.217.126:42403)
0926 07:02:30.223511 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:30.223513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:30.223573 (+ 60us) server_negotiation.cc:407] Connection header received
0926 07:02:30.223630 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:30.223631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:30.223647 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:30.223696 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:30.223966 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.224449 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.225160 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.225340 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.225387 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.227601 (+ 2214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:54439: BlockingRecv error: recv got EOF from 127.1.217.66:54439 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:30 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:30.362715 11068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:30.358265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:51823)
0926 07:02:30.358412 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:30.358421 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:30.358496 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:30.358607 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:30.358607 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:30.358788 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:30.358835 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.358837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.359476 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.359477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.359992 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:30.359993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.360294 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.360294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:30.360332 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.360755 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:30.360759 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:30.362618 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260926 07:02:30.362757 11066 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:30.358323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:51823 (local address 127.1.217.126:42403)
0926 07:02:30.358399 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:30.358401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:30.358500 (+ 99us) server_negotiation.cc:407] Connection header received
0926 07:02:30.358533 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:30.358534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:30.358546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:30.358601 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:30.358897 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.359401 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.360061 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:30.360235 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:30.360286 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:30.362695 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:51823: BlockingRecv error: recv got EOF from 127.1.217.67:51823 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:31.134051 11069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:31.129523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:37049)
0926 07:02:31.129679 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:31.129691 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:31.129788 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:31.129969 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:31.129970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:31.130138 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:31.130183 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.130184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.130816 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.130817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.131322 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.131324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.131690 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.131691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.131761 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.132220 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:31.132225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:31.133992 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
W20260926 07:02:31.134243 11070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:31.129594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:37049 (local address 127.1.217.126:42403)
0926 07:02:31.129691 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:31.129693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:31.129767 (+ 74us) server_negotiation.cc:407] Connection header received
0926 07:02:31.129842 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:31.129865 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:31.129879 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:31.129959 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:31.130244 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.130754 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.131407 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.131593 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.131689 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.134080 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:37049: BlockingRecv error: recv got EOF from 127.1.217.65:37049 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":48,"threads_started":1}
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:31.232445 11071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:31.227974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:45583)
0926 07:02:31.228098 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:31.228111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:31.228173 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:31.228321 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:31.228322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:31.228481 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:31.228524 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.228525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.229174 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.229175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.229706 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.229708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.230012 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.230012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.230051 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.230511 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:31.230522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:31.232387 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
W20260926 07:02:31.232594 11070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:31.228069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:45583 (local address 127.1.217.126:42403)
0926 07:02:31.228126 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:31.228129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:31.228160 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:02:31.228242 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:31.228242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:31.228253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:31.228307 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:31.228595 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.229115 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.229774 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.229955 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.230004 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.232516 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:45583: BlockingRecv error: recv got EOF from 127.1.217.66:45583 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:31 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.67@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:31.367817 11072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:31.363182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.67:34965)
0926 07:02:31.363337 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:31.363349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:31.363432 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:31.363588 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:31.363589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:31.363787 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:31.363833 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.363835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.364445 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.364446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.365056 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:31.365058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.365548 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.365549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:31.365646 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.366086 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:31.366090 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:31.367759 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
W20260926 07:02:31.367934 11070 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:31.363261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:34965 (local address 127.1.217.126:42403)
0926 07:02:31.363321 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:31.363323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:31.363410 (+ 87us) server_negotiation.cc:407] Connection header received
0926 07:02:31.363505 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:31.363506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:31.363517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:31.363565 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:31.363900 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.364380 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.365184 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:31.365482 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:31.365558 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:31.367846 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.67:34965: BlockingRecv error: recv got EOF from 127.1.217.67:34965 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 07:02:32 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:32 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.65@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:32.138918 11073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.134529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.65:42977)
0926 07:02:32.134665 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:32.134676 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:32.134730 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:32.134867 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:32.134868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:32.135017 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:32.135065 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.135067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.135676 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.135676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.136209 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.136211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.136642 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.136642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.136686 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.137165 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:32.137169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:32.138851 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260926 07:02:32.139045 11074 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:32.134627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:42977 (local address 127.1.217.126:42403)
0926 07:02:32.134740 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.134742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.134757 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:02:32.134778 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.134779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.134789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.134846 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.135127 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.135614 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.136277 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.136575 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.136638 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.138934 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.65:42977: BlockingRecv error: recv got EOF from 127.1.217.65:42977 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
Sep 26 07:02:32 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
Sep 26 07:02:32 dist-test-slave-t1mc krb5kdc[10016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.1.217.66@KRBTEST.COM for oryx/127.1.217.126@KRBTEST.COM, Server not found in Kerberos database
I20260926 07:02:32.237507 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.233033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42403 (local address 127.1.217.66:58419)
0926 07:02:32.233159 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:32.233170 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:32.233232 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:32.233358 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:32.233358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:32.233514 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:32.233562 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.233563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.234161 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.234161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.234706 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.234708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.235064 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.235065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.235102 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.235573 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:32.235577 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:32.237400 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.1.217.126:42403: Server oryx/127.1.217.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260926 07:02:32.237538 11074 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:32.233108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:58419 (local address 127.1.217.126:42403)
0926 07:02:32.233181 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.233183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.233215 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:02:32.233282 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.233283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.233293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.233343 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.233618 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.234099 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.234806 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.234995 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.235059 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.237474 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.217.66:58419: BlockingRecv error: recv got EOF from 127.1.217.66:58419 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:02:32.295070 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10599
I20260926 07:02:32.299572 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10751
I20260926 07:02:32.303606 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10883
I20260926 07:02:32.309388 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 10529
2026-09-26T07:02:32Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17376 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T07:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:32Z Disabled control of system clock
I20260926 07:02:32.335687 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45825
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:45825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 07:02:32.417506 11079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:32.417708 11079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:32.417728 11079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:32.418828 11079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:32.418861 11079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:32.418874 11079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:32.418886 11079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:32.418898 11079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:32.420188 11079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:45825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:45825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:32.420377 11079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:32.420595 11079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:32.425426 11084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.425408 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.425396 11085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:32.425462 11079 server_base.cc:1034] running on GCE node
I20260926 07:02:32.425841 11079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:32.426110 11079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:32.427265 11079 hybrid_clock.cc:648] HybridClock initialized: now 1790406152427257 us; error 43 us; skew 500 ppm
I20260926 07:02:32.428354 11079 webserver.cc:460] Webserver started at http://127.1.217.126:43767/ using document root <none> and password file <none>
I20260926 07:02:32.428591 11079 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:32.428653 11079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:32.428818 11079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:32.429811 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "1d73fca914064750989fc0f6934dcd6a"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "9b02eb305868d817a5264122c9357aec"
server_key_iv: "de004994979d0f8d0e2e5f3012709dd9"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.430150 11079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "1d73fca914064750989fc0f6934dcd6a"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "9b02eb305868d817a5264122c9357aec"
server_key_iv: "de004994979d0f8d0e2e5f3012709dd9"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.431349 11079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:32.432003 11093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.432163 11079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:32.432252 11079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "1d73fca914064750989fc0f6934dcd6a"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "9b02eb305868d817a5264122c9357aec"
server_key_iv: "de004994979d0f8d0e2e5f3012709dd9"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.432329 11079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:32.447916 11079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:32.448203 11079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:32.448352 11079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:32.452035 11079 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:45825
I20260926 07:02:32.452078 11145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:45825 every 8 connection(s)
I20260926 07:02:32.452410 11079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:32.453100 11146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:32.455585 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a: Bootstrap starting.
I20260926 07:02:32.456166 11146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:32.456468 11146 log.cc:824] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:32.457149 11146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a: No bootstrap required, opened a new log
I20260926 07:02:32.458648 11146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } }
I20260926 07:02:32.458779 11146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:32.458801 11146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d73fca914064750989fc0f6934dcd6a, State: Initialized, Role: FOLLOWER
I20260926 07:02:32.458967 11146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } }
I20260926 07:02:32.459024 11146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:32.459045 11146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:32.459106 11146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:32.459585 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11079
I20260926 07:02:32.459681 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:32.459734 11146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } }
I20260926 07:02:32.459842 11146 leader_election.cc:302] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d73fca914064750989fc0f6934dcd6a; no voters:
I20260926 07:02:32.459894 1893 external_mini_cluster.cc:1428] Setting key b128c11a7242f23d8f0c6b08e31f50c6
I20260926 07:02:32.459998 11146 leader_election.cc:288] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:32.460045 11149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:32.460246 11146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:32.460258 11149 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [term 1 LEADER]: Becoming Leader. State: Replica: 1d73fca914064750989fc0f6934dcd6a, State: Running, Role: LEADER
I20260926 07:02:32.460469 11149 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } }
I20260926 07:02:32.461012 11150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d73fca914064750989fc0f6934dcd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } } }
I20260926 07:02:32.461097 11150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:32.461392 11160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:32.461495 11151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d73fca914064750989fc0f6934dcd6a. Latest consensus state: current_term: 1 leader_uuid: "1d73fca914064750989fc0f6934dcd6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d73fca914064750989fc0f6934dcd6a" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 45825 } } }
I20260926 07:02:32.461632 11151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:32.461990 11160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:32.463550 11160 catalog_manager.cc:1347] Generated new cluster ID: 1ddbcfae83e84e76be2a1d9007852433
I20260926 07:02:32.463606 11160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:32.465853 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.460503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.1.217.126:45825)
0926 07:02:32.460689 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.460698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.460946 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:02:32.461175 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.461183 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.461256 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.461376 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.461746 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.463166 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.464047 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.464455 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.464584 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.465251 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:32.465254 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:32.465255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:32.465426 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:32.465458 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:32.465468 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:32.465543 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:32.465613 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:02:32.465689 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260926 07:02:32.496611 11160 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:32.497363 11160 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:32.499657 11160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d73fca914064750989fc0f6934dcd6a: Generated new TSK 0
I20260926 07:02:32.499841 11160 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:32.501946 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--builtin_ntp_servers=127.1.217.84:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:02:32.581030 11170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:32.581249 11170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:32.581282 11170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:32.582437 11170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:32.582492 11170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:32.582530 11170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:32.583841 11170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:32.584092 11170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:32.584376 11170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:32.589275 11175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:32.589366 11170 server_base.cc:1034] running on GCE node
W20260926 07:02:32.589452 11178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.589301 11176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:32.589710 11170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:32.589960 11170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:32.591158 11170 hybrid_clock.cc:648] HybridClock initialized: now 1790406152591146 us; error 29 us; skew 500 ppm
I20260926 07:02:32.592669 11170 webserver.cc:460] Webserver started at http://127.1.217.65:44515/ using document root <none> and password file <none>
I20260926 07:02:32.592931 11170 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:32.592989 11170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:32.593108 11170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:32.594470 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "0839432bd1f242cda380382fe6f7c5ab"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "244dbe407684735781fcd9e8665ad0bb"
server_key_iv: "6e7e96aecd75de71e0d9857a7a752b80"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.594823 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "0839432bd1f242cda380382fe6f7c5ab"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "244dbe407684735781fcd9e8665ad0bb"
server_key_iv: "6e7e96aecd75de71e0d9857a7a752b80"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.596001 11170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:32.596643 11184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.596813 11170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:32.596894 11170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "0839432bd1f242cda380382fe6f7c5ab"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "244dbe407684735781fcd9e8665ad0bb"
server_key_iv: "6e7e96aecd75de71e0d9857a7a752b80"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.596944 11170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:32.612012 11170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:32.612254 11170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:32.612351 11170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:32.612517 11170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:32.612828 11170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:32.612862 11170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.612883 11170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:32.612900 11170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.618799 11170 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:37169
I20260926 07:02:32.618875 11297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:37169 every 8 connection(s)
I20260926 07:02:32.619179 11170 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:32.624490 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.619906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:58283 (local address 127.1.217.126:45825)
0926 07:02:32.619951 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.619953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.620206 (+ 253us) server_negotiation.cc:407] Connection header received
0926 07:02:32.620722 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.620724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.620736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.620817 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.621456 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.621983 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.622980 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.623244 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.623355 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.624164 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:32.624165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:32.624165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:32.624170 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:32.624189 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:32.624193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:32.624252 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:32.624356 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 07:02:32.624391 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:32.624699 11300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.619876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45825 (local address 127.1.217.65:58283)
0926 07:02:32.620080 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:32.620121 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:32.620621 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:32.620896 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:32.620900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:32.620910 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:32.621365 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.621369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.622078 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.622079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.622870 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.622872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.623376 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.623377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.623514 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.623994 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:32.623994 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:32.624021 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:32.624041 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:32.624263 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:32.624268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:32.624271 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:02:32.624359 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:02:32.624476 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260926 07:02:32.625211 11298 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45825
I20260926 07:02:32.625316 11298 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:32.625490 11298 heartbeater.cc:511] Master 127.1.217.126:45825 requested a full tablet report, sending...
I20260926 07:02:32.626044 11110 ts_manager.cc:194] Registered new tserver with Master: 0839432bd1f242cda380382fe6f7c5ab (127.1.217.65:37169)
I20260926 07:02:32.626813 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11170
I20260926 07:02:32.626803 11110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.65:58283
I20260926 07:02:32.626929 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:32.627409 1893 external_mini_cluster.cc:1428] Setting key 0e67946a5cae597dabd6f3c24c70fa91
I20260926 07:02:32.628288 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--builtin_ntp_servers=127.1.217.84:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 07:02:32.712267 11301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:32.712541 11301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:32.712564 11301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:32.713763 11301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:32.713804 11301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:32.713828 11301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:32.715108 11301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:32.715291 11301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:32.715718 11301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:32.720638 11309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.720646 11306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.720647 11307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:32.720804 11301 server_base.cc:1034] running on GCE node
I20260926 07:02:32.721156 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:32.721378 11301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:32.722533 11301 hybrid_clock.cc:648] HybridClock initialized: now 1790406152722521 us; error 40 us; skew 500 ppm
I20260926 07:02:32.723696 11301 webserver.cc:460] Webserver started at http://127.1.217.66:38571/ using document root <none> and password file <none>
I20260926 07:02:32.723932 11301 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:32.723979 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:32.724131 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:32.725240 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "29246dd3b54d4d6ea77f6ea3f88f17c9"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "b9fa3e9e4c373de80ec53522fc0e3684"
server_key_iv: "1ad23af49c153a5aa437547f605261b8"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.725593 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "29246dd3b54d4d6ea77f6ea3f88f17c9"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "b9fa3e9e4c373de80ec53522fc0e3684"
server_key_iv: "1ad23af49c153a5aa437547f605261b8"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.726871 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:32.727535 11315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.727706 11301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:32.727801 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "29246dd3b54d4d6ea77f6ea3f88f17c9"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "b9fa3e9e4c373de80ec53522fc0e3684"
server_key_iv: "1ad23af49c153a5aa437547f605261b8"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.727885 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:32.741906 11301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:32.742173 11301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:32.742332 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:32.742549 11301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:32.742861 11301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:32.742913 11301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.742945 11301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:32.742975 11301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.748306 11301 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:35131
I20260926 07:02:32.748383 11428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:35131 every 8 connection(s)
I20260926 07:02:32.748703 11301 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:32.753021 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11301
I20260926 07:02:32.753120 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:32.753371 1893 external_mini_cluster.cc:1428] Setting key 93d014b4661d17c224ef1f08d6241cae
I20260926 07:02:32.753633 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.749430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:41319 (local address 127.1.217.126:45825)
0926 07:02:32.749474 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.749476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.749703 (+ 227us) server_negotiation.cc:407] Connection header received
0926 07:02:32.750160 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.750161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.750173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.750234 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.750828 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.751299 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.752198 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.752365 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.752412 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.753310 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:32.753312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:32.753312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:32.753318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:32.753332 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:32.753337 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:32.753403 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:32.753525 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 07:02:32.753556 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:02:32.753844 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.749381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45825 (local address 127.1.217.66:41319)
0926 07:02:32.749573 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:32.749612 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:32.750081 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:32.750284 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:32.750287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:32.750295 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:32.750761 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.750764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.751378 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.751379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.752128 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.752129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.752430 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.752431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.752532 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.753149 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:32.753150 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:32.753186 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:32.753214 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:32.753402 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:32.753405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:32.753407 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:02:32.753519 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 07:02:32.753616 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260926 07:02:32.754314 11429 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45825
I20260926 07:02:32.754354 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--builtin_ntp_servers=127.1.217.84:33403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 07:02:32.754400 11429 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:32.754591 11429 heartbeater.cc:511] Master 127.1.217.126:45825 requested a full tablet report, sending...
I20260926 07:02:32.754933 11110 ts_manager.cc:194] Registered new tserver with Master: 29246dd3b54d4d6ea77f6ea3f88f17c9 (127.1.217.66:35131)
I20260926 07:02:32.755285 11110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.66:41319
W20260926 07:02:32.834656 11432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:32.834833 11432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:32.834868 11432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:32.836020 11432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:32.836057 11432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:32.836081 11432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:32.837316 11432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:45825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:32.837491 11432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:32.837698 11432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:32.842434 11440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.842445 11437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:32.842514 11438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:32.842729 11432 server_base.cc:1034] running on GCE node
I20260926 07:02:32.842953 11432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:32.843173 11432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:32.844305 11432 hybrid_clock.cc:648] HybridClock initialized: now 1790406152844281 us; error 39 us; skew 500 ppm
I20260926 07:02:32.845546 11432 webserver.cc:460] Webserver started at http://127.1.217.67:41171/ using document root <none> and password file <none>
I20260926 07:02:32.845806 11432 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:32.845875 11432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:32.846005 11432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:32.847029 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "3f74da03cb82480f8a0d7204cb2a0c1f"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "d385880320e313d701e3670f6696c0c9"
server_key_iv: "57ebafdc7f60f4e06e8b0b34ebcf27e0"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.847376 11432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "3f74da03cb82480f8a0d7204cb2a0c1f"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "d385880320e313d701e3670f6696c0c9"
server_key_iv: "57ebafdc7f60f4e06e8b0b34ebcf27e0"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.848627 11432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:02:32.849346 11446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.849525 11432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.849610 11432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "3f74da03cb82480f8a0d7204cb2a0c1f"
format_stamp: "Formatted at 2026-09-26 07:02:32 on dist-test-slave-t1mc"
server_key: "d385880320e313d701e3670f6696c0c9"
server_key_iv: "57ebafdc7f60f4e06e8b0b34ebcf27e0"
server_key_version: "encryptionkey@0"
I20260926 07:02:32.849694 11432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:32.860868 11432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:32.861123 11432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:32.861268 11432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:32.861491 11432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:32.861810 11432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:32.861862 11432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.861896 11432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:32.861922 11432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:32.867395 11432 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:35087
I20260926 07:02:32.867429 11559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:35087 every 8 connection(s)
I20260926 07:02:32.867756 11432 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:32.868922 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11432
I20260926 07:02:32.868996 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:32.869179 1893 external_mini_cluster.cc:1428] Setting key f9afa2290ac939fd2bc94d254cbceae3
I20260926 07:02:32.873004 11153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.868488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:48649 (local address 127.1.217.126:45825)
0926 07:02:32.868535 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.868537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.868764 (+ 227us) server_negotiation.cc:407] Connection header received
0926 07:02:32.869424 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.869426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.869438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.869511 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.870378 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.870853 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.871773 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.871951 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.871999 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.872680 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:32.872681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:32.872681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:32.872685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:32.872697 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:32.872700 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:32.872741 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:32.872926 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:02:32.872948 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:02:32.873172 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:32.868461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:45825 (local address 127.1.217.67:48649)
0926 07:02:32.868632 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:32.868671 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:32.869306 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:32.869639 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:32.869644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:32.869653 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:32.870300 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.870305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.870929 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.870930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.871704 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:32.871706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:32.872017 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:32.872018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:32.872120 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:32.872564 (+ 444us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:32.872564 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:32.872590 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:32.872609 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:32.872741 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:32.872743 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:32.872745 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:02:32.872912 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 07:02:32.872987 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260926 07:02:32.873720 11560 heartbeater.cc:347] Connected to a master server at 127.1.217.126:45825
I20260926 07:02:32.873827 11560 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:32.874001 11560 heartbeater.cc:511] Master 127.1.217.126:45825 requested a full tablet report, sending...
I20260926 07:02:32.874378 11110 ts_manager.cc:194] Registered new tserver with Master: 3f74da03cb82480f8a0d7204cb2a0c1f (127.1.217.67:35087)
I20260926 07:02:32.874730 11110 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.67:48649
I20260926 07:02:32.881026 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:02:32.882791 11153 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:32.882334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.1.217.126:45825)
0926 07:02:32.882427 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:32.882429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:32.882463 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:02:32.882527 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:32.882528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:32.882536 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:32.882581 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:32.882725 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48692: BlockingRecv error: recv got EOF from 127.0.0.1:48692 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 07:02:32.882843 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11170
I20260926 07:02:32.887221 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11301
I20260926 07:02:32.891176 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11432
I20260926 07:02:32.895174 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11079
2026-09-26T07:02:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (585 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T07:02:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:32Z Disabled control of system clock
I20260926 07:02:32.920951 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33217
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:02:32.998556 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:32.998752 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:32.998773 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:32.999811 11571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:32.999850 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:32.999862 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:32.999876 11571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:32.999887 11571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:33.001129 11571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:33.001307 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:33.001518 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:33.006078 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.006273 11571 server_base.cc:1034] running on GCE node
W20260926 07:02:33.006108 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:33.006080 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.006592 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:33.006850 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:33.008009 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790406153007986 us; error 38 us; skew 500 ppm
I20260926 07:02:33.009261 11571 webserver.cc:460] Webserver started at http://127.1.217.126:41743/ using document root <none> and password file <none>
I20260926 07:02:33.009497 11571 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:33.009541 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:33.009706 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:33.010699 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "76173cd2a7024678b6a5f409df47a0a3"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "20df1e3b9050d2c58d607f4dbe8b3642"
server_key_iv: "97c0a783199383d1f4cefebd901f8ad2"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.011042 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "76173cd2a7024678b6a5f409df47a0a3"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "20df1e3b9050d2c58d607f4dbe8b3642"
server_key_iv: "97c0a783199383d1f4cefebd901f8ad2"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.012245 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:33.012961 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.013120 11571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:33.013235 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "76173cd2a7024678b6a5f409df47a0a3"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "20df1e3b9050d2c58d607f4dbe8b3642"
server_key_iv: "97c0a783199383d1f4cefebd901f8ad2"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.013314 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:33.016480 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:33.016731 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:33.016891 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:33.020875 11571 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33217
I20260926 07:02:33.020908 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33217 every 8 connection(s)
I20260926 07:02:33.021268 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:33.021821 11638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:33.024433 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3: Bootstrap starting.
I20260926 07:02:33.024513 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11571
I20260926 07:02:33.024637 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:33.024868 1893 external_mini_cluster.cc:1428] Setting key 0af53411ba7af8efa74a556794a11c68
I20260926 07:02:33.025175 11638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:33.025499 11638 log.cc:824] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:33.026216 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3: No bootstrap required, opened a new log
I20260926 07:02:33.027529 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.025492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.1.217.126:33217)
0926 07:02:33.025686 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:33.025695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:33.025727 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:02:33.026255 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:33.026264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:33.026314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:33.026470 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:33.027093 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:33.027096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:33.027097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:33.027101 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:33.027144 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:33.027158 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:33.027251 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:33.027280 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 07:02:33.027333 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260926 07:02:33.028122 11638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } }
I20260926 07:02:33.028291 11638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:33.028337 11638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76173cd2a7024678b6a5f409df47a0a3, State: Initialized, Role: FOLLOWER
I20260926 07:02:33.028460 11638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } }
I20260926 07:02:33.028546 11638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:33.028589 11638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:33.028637 11638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:33.029490 11638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } }
I20260926 07:02:33.029605 11638 leader_election.cc:302] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76173cd2a7024678b6a5f409df47a0a3; no voters:
I20260926 07:02:33.029759 11638 leader_election.cc:288] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:33.029829 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:33.029969 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [term 1 LEADER]: Becoming Leader. State: Replica: 76173cd2a7024678b6a5f409df47a0a3, State: Running, Role: LEADER
I20260926 07:02:33.030007 11638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:33.030068 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } }
I20260926 07:02:33.030457 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76173cd2a7024678b6a5f409df47a0a3. Latest consensus state: current_term: 1 leader_uuid: "76173cd2a7024678b6a5f409df47a0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } } }
I20260926 07:02:33.030558 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:33.030450 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76173cd2a7024678b6a5f409df47a0a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76173cd2a7024678b6a5f409df47a0a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33217 } } }
I20260926 07:02:33.030721 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:33.031010 11652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:33.031530 11652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:33.032917 11652 catalog_manager.cc:1347] Generated new cluster ID: 54b44ebe254b43548215f55b5efb7dbb
I20260926 07:02:33.032977 11652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:33.048671 11652 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:33.048898 11652 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:33.053109 11652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76173cd2a7024678b6a5f409df47a0a3: Generated new TSK 0
I20260926 07:02:33.053282 11652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:02:33.059891 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--builtin_ntp_servers=127.1.217.84:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:02:33.142307 11662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:33.142532 11662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:33.142567 11662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:33.143705 11662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:33.143766 11662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:33.143810 11662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:33.145167 11662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:33.145433 11662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:33.145717 11662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:33.151364 11668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:33.151372 11667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.151501 11662 server_base.cc:1034] running on GCE node
W20260926 07:02:33.151392 11670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.151815 11662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:33.152069 11662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:33.153259 11662 hybrid_clock.cc:648] HybridClock initialized: now 1790406153153235 us; error 41 us; skew 500 ppm
I20260926 07:02:33.154727 11662 webserver.cc:460] Webserver started at http://127.1.217.65:46393/ using document root <none> and password file <none>
I20260926 07:02:33.154981 11662 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:33.155032 11662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:33.155145 11662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:33.156401 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "cb16d90654524155a0b4dcc8fe56b369"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "0284148a4b2a571ced96a0643238cc28"
server_key_iv: "665edf25e47f6d69a3ae035252476f40"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.156679 11662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "cb16d90654524155a0b4dcc8fe56b369"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "0284148a4b2a571ced96a0643238cc28"
server_key_iv: "665edf25e47f6d69a3ae035252476f40"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.157986 11662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:02:33.158668 11676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.158826 11662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:02:33.158927 11662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "cb16d90654524155a0b4dcc8fe56b369"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "0284148a4b2a571ced96a0643238cc28"
server_key_iv: "665edf25e47f6d69a3ae035252476f40"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.159014 11662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:33.162292 11662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:33.162562 11662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:33.162703 11662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:33.162920 11662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:33.163209 11662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:33.163261 11662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.163300 11662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:33.163327 11662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.169137 11662 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:44939
I20260926 07:02:33.169173 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:44939 every 8 connection(s)
I20260926 07:02:33.169493 11662 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:02:33.172361 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.170186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:54817 (local address 127.1.217.126:33217)
0926 07:02:33.170227 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:33.170229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:33.170460 (+ 231us) server_negotiation.cc:407] Connection header received
0926 07:02:33.170956 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:33.170957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:33.170965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:33.171011 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:33.172101 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:33.172103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:33.172104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:33.172109 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:33.172124 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:33.172129 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:33.172181 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:33.172280 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:02:33.172295 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:33.172544 11792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.170170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33217 (local address 127.1.217.65:54817)
0926 07:02:33.170341 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:33.170373 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:33.170843 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:33.171096 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:33.171099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:33.171106 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:33.171946 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:33.171946 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:33.171973 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:33.171999 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:33.172177 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:33.172181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:33.172185 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:33.172289 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 07:02:33.172347 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260926 07:02:33.173046 11790 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33217
I20260926 07:02:33.173117 11790 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:33.173277 11790 heartbeater.cc:511] Master 127.1.217.126:33217 requested a full tablet report, sending...
I20260926 07:02:33.173719 11602 ts_manager.cc:194] Registered new tserver with Master: cb16d90654524155a0b4dcc8fe56b369 (127.1.217.65:44939)
I20260926 07:02:33.177356 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11662
I20260926 07:02:33.177460 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:33.177637 1893 external_mini_cluster.cc:1428] Setting key 28ae3ea061007d36c7bc8a4e1812e602
I20260926 07:02:33.178495 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--builtin_ntp_servers=127.1.217.84:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:02:33.257092 11793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:33.257320 11793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:33.257356 11793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:33.258500 11793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:33.258566 11793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:33.258611 11793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:33.259932 11793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:33.260185 11793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:33.260469 11793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:33.265442 11801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:33.265435 11798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.265726 11793 server_base.cc:1034] running on GCE node
W20260926 07:02:33.265807 11799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.266055 11793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:33.266302 11793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:33.267458 11793 hybrid_clock.cc:648] HybridClock initialized: now 1790406153267447 us; error 27 us; skew 500 ppm
I20260926 07:02:33.268608 11793 webserver.cc:460] Webserver started at http://127.1.217.66:45373/ using document root <none> and password file <none>
I20260926 07:02:33.268918 11793 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:33.268987 11793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:33.269120 11793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:33.270145 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "d0828f8152e046f19e4e6dcd2746229e"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "c1c0eb6ddd9da9b8ce69042a84d33bdb"
server_key_iv: "5d4201fe99b1395d9bc2549207bd190e"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.270514 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "d0828f8152e046f19e4e6dcd2746229e"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "c1c0eb6ddd9da9b8ce69042a84d33bdb"
server_key_iv: "5d4201fe99b1395d9bc2549207bd190e"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.271761 11793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:33.272418 11807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.272593 11793 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:33.272683 11793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "d0828f8152e046f19e4e6dcd2746229e"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "c1c0eb6ddd9da9b8ce69042a84d33bdb"
server_key_iv: "5d4201fe99b1395d9bc2549207bd190e"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.272799 11793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:33.275974 11793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:33.276222 11793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:33.276353 11793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:33.276554 11793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:33.276909 11793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:33.276959 11793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.276997 11793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:33.277026 11793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.282970 11793 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:35723
I20260926 07:02:33.283016 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:35723 every 8 connection(s)
I20260926 07:02:33.283313 11793 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:33.286113 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.283923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:35247 (local address 127.1.217.126:33217)
0926 07:02:33.283966 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:33.283968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:33.284216 (+ 248us) server_negotiation.cc:407] Connection header received
0926 07:02:33.284711 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:33.284712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:33.284718 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:33.284816 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:33.285885 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:33.285886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:33.285887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:33.285891 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:33.285906 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:33.285910 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:33.285958 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:33.286028 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 07:02:33.286043 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:33.286304 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.283924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33217 (local address 127.1.217.66:35247)
0926 07:02:33.284103 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:33.284134 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:33.284611 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:33.284884 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:33.284888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:33.284899 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:33.285730 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:33.285731 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:33.285764 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:33.285791 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:33.285955 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:33.285959 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:33.285963 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:33.286035 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 07:02:33.286093 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260926 07:02:33.286770 11921 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33217
I20260926 07:02:33.286885 11921 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:33.287029 11921 heartbeater.cc:511] Master 127.1.217.126:33217 requested a full tablet report, sending...
I20260926 07:02:33.287384 11602 ts_manager.cc:194] Registered new tserver with Master: d0828f8152e046f19e4e6dcd2746229e (127.1.217.66:35723)
I20260926 07:02:33.292572 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11793
I20260926 07:02:33.292649 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:33.292852 1893 external_mini_cluster.cc:1428] Setting key ebeac147f7b78392e4432e00aef911f1
I20260926 07:02:33.293808 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--builtin_ntp_servers=127.1.217.84:36577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 07:02:33.372252 11924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:33.372494 11924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:33.372534 11924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:33.373677 11924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:33.373733 11924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:33.373773 11924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:33.375164 11924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:33.375399 11924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:33.375659 11924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:33.380553 11930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:33.380564 11929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.380610 11924 server_base.cc:1034] running on GCE node
W20260926 07:02:33.380581 11932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:33.380940 11924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:33.381170 11924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:33.382329 11924 hybrid_clock.cc:648] HybridClock initialized: now 1790406153382318 us; error 27 us; skew 500 ppm
I20260926 07:02:33.383554 11924 webserver.cc:460] Webserver started at http://127.1.217.67:36305/ using document root <none> and password file <none>
I20260926 07:02:33.383777 11924 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:33.383844 11924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:33.384012 11924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:33.385107 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "18c304ec8da946009a24a45ae305ad27"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "7220b5ab22cc62edaeac562dacb9de27"
server_key_iv: "da076104744da4fd0f35ddf0f240d5c6"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.385458 11924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "18c304ec8da946009a24a45ae305ad27"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "7220b5ab22cc62edaeac562dacb9de27"
server_key_iv: "da076104744da4fd0f35ddf0f240d5c6"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.386670 11924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:33.387398 11938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.387588 11924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:33.387678 11924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "18c304ec8da946009a24a45ae305ad27"
format_stamp: "Formatted at 2026-09-26 07:02:33 on dist-test-slave-t1mc"
server_key: "7220b5ab22cc62edaeac562dacb9de27"
server_key_iv: "da076104744da4fd0f35ddf0f240d5c6"
server_key_version: "encryptionkey@0"
I20260926 07:02:33.387756 11924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:33.391006 11924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:33.391245 11924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:33.391381 11924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:33.391574 11924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:33.391870 11924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:33.391919 11924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.391958 11924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:33.391985 11924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:33.397585 11924 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:41079
I20260926 07:02:33.397619 12051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:41079 every 8 connection(s)
I20260926 07:02:33.397948 11924 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:33.400604 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.398593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:52255 (local address 127.1.217.126:33217)
0926 07:02:33.398656 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:33.398658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:33.398894 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:02:33.399304 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:33.399305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:33.399309 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:33.399359 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:33.400438 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:33.400439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:33.400439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:02:33.400441 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:33.400450 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:02:33.400454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:33.400491 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:33.400545 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 07:02:33.400556 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:02:33.400831 12054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:33.398582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33217 (local address 127.1.217.67:52255)
0926 07:02:33.398779 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:33.398809 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:33.399229 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:33.399407 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:33.399411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:33.399417 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:33.400320 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:02:33.400320 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:33.400346 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:02:33.400374 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:33.400492 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:33.400494 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:33.400496 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:02:33.400544 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:02:33.400587 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":98,"threads_started":1}
I20260926 07:02:33.401273 12052 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33217
I20260926 07:02:33.401342 12052 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:33.401501 12052 heartbeater.cc:511] Master 127.1.217.126:33217 requested a full tablet report, sending...
I20260926 07:02:33.401854 11602 ts_manager.cc:194] Registered new tserver with Master: 18c304ec8da946009a24a45ae305ad27 (127.1.217.67:41079)
I20260926 07:02:33.407953 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 11924
I20260926 07:02:33.408032 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:33.408170 1893 external_mini_cluster.cc:1428] Setting key 580a9f8108e648c784867c078693f40d
I20260926 07:02:33.408993 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 07:02:33.410673 11641 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:02:33.410260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.1.217.126:33217)
0926 07:02:33.410328 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:33.410330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:33.410399 (+ 69us) server_negotiation.cc:407] Connection header received
0926 07:02:33.410438 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:33.410439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:33.410443 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:33.410489 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:33.410613 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43356: BlockingRecv error: recv got EOF from 127.0.0.1:43356 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:02:33.410758 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11662
I20260926 07:02:33.414978 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11793
I20260926 07:02:33.418949 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11924
I20260926 07:02:33.423079 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 11571
2026-09-26T07:02:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (527 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:33 dist-test-slave-t1mc krb5kdc[12062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:33 dist-test-slave-t1mc krb5kdc[12062](info): set up 2 sockets
Sep 26 07:02:33 dist-test-slave-t1mc krb5kdc[12062](info): commencing operation
krb5kdc: starting...
W20260926 07:02:34.453569 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:02:34.571162 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43939
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:34.649340 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:34.649664 12075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:34.649760 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:34.650950 12075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:02:34.651026 12075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:34.651048 12075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:34.651067 12075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:02:34.651084 12075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:02:34.652557 12075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:34.652851 12075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:34.653075 12075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:34.657686 12081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:34.657730 12080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:34.657732 12075 server_base.cc:1034] running on GCE node
W20260926 07:02:34.657696 12083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:34.658085 12075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:34.658383 12075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:34.659525 12075 hybrid_clock.cc:648] HybridClock initialized: now 1790406154659513 us; error 28 us; skew 500 ppm
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:34.667626 12075 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:02:34.668041 12075 webserver.cc:460] Webserver started at http://127.1.217.126:42291/ using document root <none> and password file <none>
I20260926 07:02:34.668254 12075 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:34.668323 12075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:34.668463 12075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:34.669560 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d8a6717138b14f089eaa2389c288b108"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "9fdb942178d09082e4b961d57a9eb93d"
server_key_iv: "44c91706c139a61814398d58a28807ae"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.669902 12075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d8a6717138b14f089eaa2389c288b108"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "9fdb942178d09082e4b961d57a9eb93d"
server_key_iv: "44c91706c139a61814398d58a28807ae"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.671175 12075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:34.671875 12090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:34.672065 12075 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:34.672189 12075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d8a6717138b14f089eaa2389c288b108"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "9fdb942178d09082e4b961d57a9eb93d"
server_key_iv: "44c91706c139a61814398d58a28807ae"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.672261 12075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:34.687076 12075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:34.689266 12075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:34.689419 12075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:34.693063 12075 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43939
I20260926 07:02:34.693092 12142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43939 every 8 connection(s)
I20260926 07:02:34.693415 12075 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:02:34.694334 12143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:34.695598 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 12075
I20260926 07:02:34.695709 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:02:34.695889 1893 external_mini_cluster.cc:1428] Setting key b5f1be0b52fabaa8ce934bff50b49317
I20260926 07:02:34.696877 12143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: Bootstrap starting.
I20260926 07:02:34.697485 12143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:34.697813 12143 log.cc:824] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:34.698588 12143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: No bootstrap required, opened a new log
I20260926 07:02:34.700675 12143 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } }
I20260926 07:02:34.700871 12143 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:34.700924 12143 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a6717138b14f089eaa2389c288b108, State: Initialized, Role: FOLLOWER
I20260926 07:02:34.701051 12143 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } }
I20260926 07:02:34.701139 12143 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:02:34.701162 12143 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:02:34.701231 12143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:34.701777 12143 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } }
I20260926 07:02:34.701893 12143 leader_election.cc:302] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a6717138b14f089eaa2389c288b108; no voters:
I20260926 07:02:34.702090 12143 leader_election.cc:288] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:02:34.702109 12148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:34.702217 12148 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [term 1 LEADER]: Becoming Leader. State: Replica: d8a6717138b14f089eaa2389c288b108, State: Running, Role: LEADER
I20260926 07:02:34.702296 12148 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } }
I20260926 07:02:34.702354 12143 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:02:34.702632 12148 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a6717138b14f089eaa2389c288b108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } } }
I20260926 07:02:34.702701 12148 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [sys.catalog]: This master's current role is: LEADER
I20260926 07:02:34.702840 12148 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a6717138b14f089eaa2389c288b108. Latest consensus state: current_term: 1 leader_uuid: "d8a6717138b14f089eaa2389c288b108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a6717138b14f089eaa2389c288b108" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43939 } } }
I20260926 07:02:34.702910 12148 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108 [sys.catalog]: This master's current role is: LEADER
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
W20260926 07:02:34.704115 12164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:02:34.704170 12164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:02:34.704226 12153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:02:34.704423 12153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:02:34.706362 12153 catalog_manager.cc:1347] Generated new cluster ID: 2612bd02141446e595d77d52093756cb
I20260926 07:02:34.706413 12153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:02:34.707363 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:34.696540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.1.217.126:43939)
0926 07:02:34.696721 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:34.696729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:34.696753 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:02:34.697253 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:34.697260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:34.697309 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:34.697424 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:34.698223 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.699215 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.700296 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.700696 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.700838 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:34.703579 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:34.703583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:34.703584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:34.703595 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:34.705668 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:34.706085 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:34.706088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:34.706088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:34.706186 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:34.706422 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:34.706423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:34.706424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:34.706719 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:34.706884 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:34.707010 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 07:02:34.707148 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260926 07:02:34.718832 12153 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:02:34.719331 12153 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:02:34.724501 12153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: Generated new TSK 0
I20260926 07:02:34.724665 12153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:02:34.755373 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:34.834949 12169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:34.835152 12169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:34.835175 12169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:34.836318 12169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:34.836359 12169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:34.836408 12169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:02:34.837857 12169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:34.838091 12169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:34.838321 12169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:34.843308 12177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:34.843308 12174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:34.843333 12175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:34.843937 12169 server_base.cc:1034] running on GCE node
I20260926 07:02:34.844182 12169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:34.844434 12169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:34.845590 12169 hybrid_clock.cc:648] HybridClock initialized: now 1790406154845589 us; error 60 us; skew 500 ppm
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:34.848043 12169 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:02:34.848480 12169 webserver.cc:460] Webserver started at http://127.1.217.65:33507/ using document root <none> and password file <none>
I20260926 07:02:34.848719 12169 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:34.848812 12169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:34.848959 12169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:34.849998 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "92184c0545a74172901bdab88ae430ab"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "547d785b14f2787951c188acf21476ef"
server_key_iv: "1e1bab11dbed0df20602f3f8dedb6e64"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.850358 12169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "92184c0545a74172901bdab88ae430ab"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "547d785b14f2787951c188acf21476ef"
server_key_iv: "1e1bab11dbed0df20602f3f8dedb6e64"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.851676 12169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:02:34.852416 12184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:34.852604 12169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:02:34.852707 12169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "92184c0545a74172901bdab88ae430ab"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "547d785b14f2787951c188acf21476ef"
server_key_iv: "1e1bab11dbed0df20602f3f8dedb6e64"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.852808 12169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:34.865197 12169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:34.866945 12169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:34.867106 12169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:34.867338 12169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:34.867643 12169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:34.867693 12169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:34.867743 12169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:34.867771 12169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:34.873384 12169 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:46839
I20260926 07:02:34.873422 12297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:46839 every 8 connection(s)
I20260926 07:02:34.873747 12169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:34.879956 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 12169
I20260926 07:02:34.880054 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:02:34.880267 1893 external_mini_cluster.cc:1428] Setting key 7e5752713ed852537beba286d83e5cc5
I20260926 07:02:34.883358 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:34.874525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:34017 (local address 127.1.217.126:43939)
0926 07:02:34.874593 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:34.874595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:34.874840 (+ 245us) server_negotiation.cc:407] Connection header received
0926 07:02:34.875381 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:34.875382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:34.875391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:34.875445 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:34.876481 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.877008 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.877952 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.878142 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.878190 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:34.880439 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:34.880441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:34.880441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:34.880449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:34.881990 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:34.882423 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:34.882425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:34.882425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:34.882483 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:34.882737 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:34.882738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:34.882738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:34.882845 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:34.882919 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:34.883181 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 07:02:34.883262 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:02:34.883664 12300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:34.874511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43939 (local address 127.1.217.65:34017)
0926 07:02:34.874683 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:34.874719 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:34.875295 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:34.875515 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:34.875518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:34.876216 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:34.876404 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:34.876409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.877081 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.877082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:34.877880 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:34.877882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:34.878209 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:34.878210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:34.878304 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:34.878819 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:34.878827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:34.880282 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:34.882091 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:34.882098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:34.882100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:34.882334 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:34.882550 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:34.882551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:34.882551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:34.882664 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:34.882910 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:34.882916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:34.883052 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 07:02:34.883161 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:02:34.883322 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260926 07:02:34.884183 12298 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43939
I20260926 07:02:34.884291 12298 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:34.884464 12298 heartbeater.cc:511] Master 127.1.217.126:43939 requested a full tablet report, sending...
I20260926 07:02:34.885022 12107 ts_manager.cc:194] Registered new tserver with Master: 92184c0545a74172901bdab88ae430ab (127.1.217.65:46839)
I20260926 07:02:34.886004 12107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:34017
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:02:34.905239 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:34.983189 12303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:34.983428 12303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:34.983465 12303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:34.984587 12303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:34.984650 12303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:34.984695 12303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:02:34.986130 12303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:34.986410 12303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:34.986634 12303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:34.991641 12311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:34.991657 12308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:34.991686 12303 server_base.cc:1034] running on GCE node
W20260926 07:02:34.991652 12309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:34.992013 12303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:34.992194 12303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:34.993327 12303 hybrid_clock.cc:648] HybridClock initialized: now 1790406154993311 us; error 33 us; skew 500 ppm
Sep 26 07:02:34 dist-test-slave-t1mc krb5kdc[12062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:34.995656 12303 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:02:34.996104 12303 webserver.cc:460] Webserver started at http://127.1.217.66:34495/ using document root <none> and password file <none>
I20260926 07:02:34.996331 12303 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:34.996400 12303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:34.996546 12303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:34.997637 12303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "d1f77df9a0d34ea68b9766d28e097414"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "7b049d268445ef94d4efec2746f0f3a3"
server_key_iv: "8f41a8572f8c5818c2c2cce07876f489"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.997987 12303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f77df9a0d34ea68b9766d28e097414"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "7b049d268445ef94d4efec2746f0f3a3"
server_key_iv: "8f41a8572f8c5818c2c2cce07876f489"
server_key_version: "encryptionkey@0"
I20260926 07:02:34.999258 12303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:34.999972 12318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.000151 12303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:35.000243 12303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "d1f77df9a0d34ea68b9766d28e097414"
format_stamp: "Formatted at 2026-09-26 07:02:34 on dist-test-slave-t1mc"
server_key: "7b049d268445ef94d4efec2746f0f3a3"
server_key_iv: "8f41a8572f8c5818c2c2cce07876f489"
server_key_version: "encryptionkey@0"
I20260926 07:02:35.000329 12303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:35.019461 12303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:35.021787 12303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:35.021950 12303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:35.022183 12303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:35.022488 12303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:35.022542 12303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.022583 12303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:35.022614 12303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.028390 12303 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:35123
I20260926 07:02:35.028439 12431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:35123 every 8 connection(s)
I20260926 07:02:35.028745 12303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:02:35.029773 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 12303
I20260926 07:02:35.029867 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:02:35.030063 1893 external_mini_cluster.cc:1428] Setting key 512eb70cae6fc5befec5c60d6cdad989
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406154, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
I20260926 07:02:35.039593 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.029448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:46707 (local address 127.1.217.126:43939)
0926 07:02:35.029495 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.029497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.029793 (+ 296us) server_negotiation.cc:407] Connection header received
0926 07:02:35.030501 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.030502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.030513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.030573 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:35.031867 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.032340 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.033864 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.034046 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.034101 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.036670 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:35.036672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:35.036672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:35.036679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:35.038117 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.038626 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.038627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.038628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.038674 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.038927 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.038928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.038928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.039023 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:35.039073 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.039456 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.039525 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:02:35.039789 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.029437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43939 (local address 127.1.217.66:46707)
0926 07:02:35.029635 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.029686 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.030391 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.030655 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.030658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.031480 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:35.031773 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.031779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.032415 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.032416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.033777 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.033780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.034129 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.034130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.034285 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.034844 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:35.034855 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:35.036569 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:35.038202 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:35.038209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:35.038212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:35.038550 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:35.038736 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:35.038736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:35.038737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:35.038866 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:35.039132 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:35.039139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:35.039310 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.039433 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.039561 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
I20260926 07:02:35.040325 12432 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43939
I20260926 07:02:35.040426 12432 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:35.040612 12432 heartbeater.cc:511] Master 127.1.217.126:43939 requested a full tablet report, sending...
I20260926 07:02:35.041039 12107 ts_manager.cc:194] Registered new tserver with Master: d1f77df9a0d34ea68b9766d28e097414 (127.1.217.66:35123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
I20260926 07:02:35.041510 12107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:46707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:02:35.055620 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:02:35.134829 12437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:02:35.135030 12437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:02:35.135053 12437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:02:35.136210 12437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:02:35.136250 12437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:02:35.136277 12437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:02:35.137717 12437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:02:35.137953 12437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:02:35.138185 12437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:02:35.143061 12443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:02:35.143097 12442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:35.143132 12437 server_base.cc:1034] running on GCE node
W20260926 07:02:35.143112 12445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:02:35.143399 12437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:02:35.143644 12437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:02:35.144835 12437 hybrid_clock.cc:648] HybridClock initialized: now 1790406155144823 us; error 45 us; skew 500 ppm
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:02:35.147159 12437 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:02:35.147568 12437 webserver.cc:460] Webserver started at http://127.1.217.67:37915/ using document root <none> and password file <none>
I20260926 07:02:35.147809 12437 fs_manager.cc:359] Metadata directory not provided
I20260926 07:02:35.147876 12437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:02:35.148025 12437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:02:35.149113 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "4f32dd01e0d94673838da61bb0c9f038"
format_stamp: "Formatted at 2026-09-26 07:02:35 on dist-test-slave-t1mc"
server_key: "c17f03a1680c1b952cc410ded0bd801a"
server_key_iv: "a8d083d4feae584775eded9c61a748ff"
server_key_version: "encryptionkey@0"
I20260926 07:02:35.149461 12437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "4f32dd01e0d94673838da61bb0c9f038"
format_stamp: "Formatted at 2026-09-26 07:02:35 on dist-test-slave-t1mc"
server_key: "c17f03a1680c1b952cc410ded0bd801a"
server_key_iv: "a8d083d4feae584775eded9c61a748ff"
server_key_version: "encryptionkey@0"
I20260926 07:02:35.150746 12437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:02:35.151425 12452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.151590 12437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:02:35.151687 12437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "4f32dd01e0d94673838da61bb0c9f038"
format_stamp: "Formatted at 2026-09-26 07:02:35 on dist-test-slave-t1mc"
server_key: "c17f03a1680c1b952cc410ded0bd801a"
server_key_iv: "a8d083d4feae584775eded9c61a748ff"
server_key_version: "encryptionkey@0"
I20260926 07:02:35.151770 12437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:02:35.165335 12437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:02:35.167742 12437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:02:35.167865 12437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:02:35.168077 12437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:02:35.168402 12437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:02:35.168435 12437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.168463 12437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:02:35.168481 12437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:02:35.175000 12437 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:33935
I20260926 07:02:35.175076 12565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:33935 every 8 connection(s)
I20260926 07:02:35.175365 12437 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:02:35.180393 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 12437
I20260926 07:02:35.180496 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:02:35.180716 1893 external_mini_cluster.cc:1428] Setting key eb55298b422631bf06ee3af4fa97aa30
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406155, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:02:35.184736 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.176069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:49879 (local address 127.1.217.126:43939)
0926 07:02:35.176143 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.176146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.176367 (+ 221us) server_negotiation.cc:407] Connection header received
0926 07:02:35.176877 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.176878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.176888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.176939 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:35.177811 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.178285 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.179208 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.179384 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.179439 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.181954 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:35.181956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:35.181956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:35.181965 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:35.183376 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.183799 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.183800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.183800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.183857 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.184119 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.184119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.184120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.184230 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:35.184298 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.184564 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.184644 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:02:35.184939 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.176045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43939 (local address 127.1.217.67:49879)
0926 07:02:35.176250 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.176287 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.176745 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.176991 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.176994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.177547 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:02:35.177738 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.177742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.178360 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.178361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.179134 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.179135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.179492 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.179494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.179630 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.180091 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:02:35.180098 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:02:35.181866 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:02:35.183477 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:35.183483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:35.183485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:35.183708 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:35.183924 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:02:35.183925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:02:35.183925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:02:35.184036 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:02:35.184301 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:02:35.184307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:02:35.184439 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.184544 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.184695 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":124,"threads_started":1}
I20260926 07:02:35.185360 12566 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43939
I20260926 07:02:35.185464 12566 heartbeater.cc:464] Registering TS with master...
I20260926 07:02:35.185631 12566 heartbeater.cc:511] Master 127.1.217.126:43939 requested a full tablet report, sending...
I20260926 07:02:35.185976 12107 ts_manager.cc:194] Registered new tserver with Master: 4f32dd01e0d94673838da61bb0c9f038 (127.1.217.67:33935)
I20260926 07:02:35.186345 12107 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:49879
I20260926 07:02:35.192351 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:02:35.199630 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.193613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.1.217.126:43939)
0926 07:02:35.193669 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.193671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.193790 (+ 119us) server_negotiation.cc:407] Connection header received
0926 07:02:35.193863 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.193864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.193874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.193943 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:02:35.194492 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.195027 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.195747 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.195937 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.196018 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.197255 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:02:35.197257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:02:35.197257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:02:35.197264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:02:35.198509 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.198848 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.198849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.198849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.198908 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:02:35.199094 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:02:35.199095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:02:35.199095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:02:35.199240 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:02:35.199340 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.199459 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.199531 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:02:35.201001 12107 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:02:35.202744 12107 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:02:35.209018 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.204985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:46839 (local address 127.0.0.1:45534)
0926 07:02:35.205283 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.205295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.205344 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.205848 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.205849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.205851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:35.205905 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.205906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.206784 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.206785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.207874 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.207876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.208698 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.208699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.208793 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.208807 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.208883 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.208941 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":65,"thread_start_us":82,"threads_started":1}
I20260926 07:02:35.209033 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.205486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.1.217.65:46839)
0926 07:02:35.205700 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.205705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.205714 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:02:35.205751 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.205753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.205770 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.205852 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:35.205991 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.206688 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.207965 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.208612 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.208685 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.208738 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.208917 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.208956 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":67,"threads_started":1}
I20260926 07:02:35.209281 12575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.204983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:33935 (local address 127.0.0.1:53468)
0926 07:02:35.205111 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.205147 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.205209 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.205813 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.205817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.205820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:35.205886 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.205888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.206775 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.206776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.208083 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.208086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.209086 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.209088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.209161 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.209165 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.209192 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.209225 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":32,"thread_start_us":71,"threads_started":1}
I20260926 07:02:35.209563 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.205312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.1.217.67:33935)
0926 07:02:35.205642 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.205645 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.205653 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:02:35.205683 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.205684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.205719 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.205802 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:35.205950 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.206679 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.208330 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.208999 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.209337 (+ 338us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.209406 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.209431 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.209470 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":76,"threads_started":1}
I20260926 07:02:35.210158 12232 tablet_service.cc:1467] Processing CreateTablet for tablet 19150bed1bdc45a2ac04813fd6019028 (DEFAULT_TABLE table=test-table [id=9c841f726e6c46a0bfd408da685dd456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:35.210212 12500 tablet_service.cc:1467] Processing CreateTablet for tablet 19150bed1bdc45a2ac04813fd6019028 (DEFAULT_TABLE table=test-table [id=9c841f726e6c46a0bfd408da685dd456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:35.210505 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.205188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:35123 (local address 127.0.0.1:42966)
0926 07:02:35.205876 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.205887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.205941 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.207002 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.207003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.207005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:35.207039 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.207040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.208257 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.208258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.209295 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.209297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.210280 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.210281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.210339 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.210343 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.210392 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.210427 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":59,"threads_started":1}
I20260926 07:02:35.210529 12232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19150bed1bdc45a2ac04813fd6019028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:35.210532 12500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19150bed1bdc45a2ac04813fd6019028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:35.210718 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.205307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.1.217.66:35123)
0926 07:02:35.205467 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.205472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.206799 (+ 1327us) server_negotiation.cc:407] Connection header received
0926 07:02:35.206842 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.206844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.206859 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.206926 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:35.207109 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.208157 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.209388 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.210121 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.210194 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.210246 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.210599 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.210644 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260926 07:02:35.213111 12580 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Bootstrap starting.
I20260926 07:02:35.213804 12580 tablet_bootstrap.cc:654] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:35.213898 12366 tablet_service.cc:1467] Processing CreateTablet for tablet 19150bed1bdc45a2ac04813fd6019028 (DEFAULT_TABLE table=test-table [id=9c841f726e6c46a0bfd408da685dd456]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:02:35.214247 12366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19150bed1bdc45a2ac04813fd6019028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:02:35.214403 12580 log.cc:824] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:35.214727 12581 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: Bootstrap starting.
I20260926 07:02:35.215094 12580 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: No bootstrap required, opened a new log
I20260926 07:02:35.215175 12580 ts_tablet_manager.cc:1397] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:02:35.215359 12581 tablet_bootstrap.cc:654] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:35.215885 12581 log.cc:824] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:35.216650 12581 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: No bootstrap required, opened a new log
I20260926 07:02:35.216722 12581 ts_tablet_manager.cc:1397] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:35.216907 12580 raft_consensus.cc:348] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.217096 12580 raft_consensus.cc:374] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:35.217137 12580 raft_consensus.cc:729] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f32dd01e0d94673838da61bb0c9f038, State: Initialized, Role: FOLLOWER
I20260926 07:02:35.217258 12580 consensus_queue.cc:260] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.217535 12566 heartbeater.cc:503] Master 127.1.217.126:43939 was elected leader, sending a full tablet report...
I20260926 07:02:35.217795 12580 ts_tablet_manager.cc:1428] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:02:35.217851 12585 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Bootstrap starting.
I20260926 07:02:35.218420 12585 tablet_bootstrap.cc:654] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Neither blocks nor log segments found. Creating new log.
I20260926 07:02:35.218883 12581 raft_consensus.cc:348] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.219019 12585 log.cc:824] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Log is configured to *not* fsync() on all Append() calls
I20260926 07:02:35.219045 12581 raft_consensus.cc:374] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:35.219090 12581 raft_consensus.cc:729] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92184c0545a74172901bdab88ae430ab, State: Initialized, Role: FOLLOWER
I20260926 07:02:35.219216 12581 consensus_queue.cc:260] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.219496 12581 ts_tablet_manager.cc:1428] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:02:35.219511 12298 heartbeater.cc:503] Master 127.1.217.126:43939 was elected leader, sending a full tablet report...
I20260926 07:02:35.219707 12585 tablet_bootstrap.cc:492] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: No bootstrap required, opened a new log
I20260926 07:02:35.219775 12585 ts_tablet_manager.cc:1397] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:02:35.221596 12585 raft_consensus.cc:348] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.221719 12585 raft_consensus.cc:374] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:02:35.221740 12585 raft_consensus.cc:729] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1f77df9a0d34ea68b9766d28e097414, State: Initialized, Role: FOLLOWER
I20260926 07:02:35.221841 12585 consensus_queue.cc:260] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.222072 12585 ts_tablet_manager.cc:1428] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:02:35.222134 12432 heartbeater.cc:503] Master 127.1.217.126:43939 was elected leader, sending a full tablet report...
I20260926 07:02:35.235224 12588 raft_consensus.cc:482] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:02:35.235319 12588 raft_consensus.cc:504] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.235715 12588 leader_election.cc:288] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f32dd01e0d94673838da61bb0c9f038 (127.1.217.67:33935), 92184c0545a74172901bdab88ae430ab (127.1.217.65:46839)
I20260926 07:02:35.239599 12578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.235986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:55377 (local address 127.1.217.67:33935)
0926 07:02:35.236028 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.236030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.236127 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:02:35.236176 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.236177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.236186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.236241 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:35.236461 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.237149 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.238690 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.239251 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.239326 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.239334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.239488 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.239523 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:02:35.239589 12579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.236080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:46615 (local address 127.1.217.65:46839)
0926 07:02:35.236143 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.236145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.236152 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:35.236176 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.236177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.236186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.236238 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:02:35.236359 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.237126 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.238690 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.239252 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.239326 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.239334 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.239488 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.239523 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:35.239609 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.235862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:46839 (local address 127.1.217.66:46615)
0926 07:02:35.235958 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.235969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.236011 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.236256 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.236257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.236258 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:35.236313 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.236315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.237219 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.237220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.238604 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.238607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.239331 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.239332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.239408 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.239414 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.239470 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.239513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":128}
I20260926 07:02:35.239614 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.235898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:33935 (local address 127.1.217.66:55377)
0926 07:02:35.236047 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.236059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:02:35.236121 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:02:35.236365 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:02:35.236366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:02:35.236367 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:02:35.236404 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.236405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.237219 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.237220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.238604 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:02:35.238607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.239331 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.239332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:02:35.239408 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.239414 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:02:35.239470 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:02:35.239513 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":64,"thread_start_us":105,"threads_started":1}
I20260926 07:02:35.239938 12252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19150bed1bdc45a2ac04813fd6019028" candidate_uuid: "d1f77df9a0d34ea68b9766d28e097414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92184c0545a74172901bdab88ae430ab" is_pre_election: true
I20260926 07:02:35.240106 12252 raft_consensus.cc:2454] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f77df9a0d34ea68b9766d28e097414 in term 0.
I20260926 07:02:35.240175 12520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19150bed1bdc45a2ac04813fd6019028" candidate_uuid: "d1f77df9a0d34ea68b9766d28e097414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f32dd01e0d94673838da61bb0c9f038" is_pre_election: true
I20260926 07:02:35.240306 12520 raft_consensus.cc:2454] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1f77df9a0d34ea68b9766d28e097414 in term 0.
I20260926 07:02:35.240317 12320 leader_election.cc:302] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92184c0545a74172901bdab88ae430ab, d1f77df9a0d34ea68b9766d28e097414; no voters:
I20260926 07:02:35.240430 12588 raft_consensus.cc:2781] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:02:35.240499 12588 raft_consensus.cc:482] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:02:35.240527 12588 raft_consensus.cc:3037] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:35.241367 12588 raft_consensus.cc:504] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.241535 12588 leader_election.cc:288] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [CANDIDATE]: Term 1 election: Requested vote from peers 4f32dd01e0d94673838da61bb0c9f038 (127.1.217.67:33935), 92184c0545a74172901bdab88ae430ab (127.1.217.65:46839)
I20260926 07:02:35.241667 12252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19150bed1bdc45a2ac04813fd6019028" candidate_uuid: "d1f77df9a0d34ea68b9766d28e097414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92184c0545a74172901bdab88ae430ab"
I20260926 07:02:35.241667 12520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19150bed1bdc45a2ac04813fd6019028" candidate_uuid: "d1f77df9a0d34ea68b9766d28e097414" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f32dd01e0d94673838da61bb0c9f038"
I20260926 07:02:35.241739 12252 raft_consensus.cc:3037] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:35.241765 12520 raft_consensus.cc:3037] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:02:35.242611 12520 raft_consensus.cc:2454] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f77df9a0d34ea68b9766d28e097414 in term 1.
I20260926 07:02:35.242611 12252 raft_consensus.cc:2454] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1f77df9a0d34ea68b9766d28e097414 in term 1.
I20260926 07:02:35.242770 12320 leader_election.cc:302] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92184c0545a74172901bdab88ae430ab, d1f77df9a0d34ea68b9766d28e097414; no voters:
I20260926 07:02:35.242859 12588 raft_consensus.cc:2781] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:02:35.242990 12588 raft_consensus.cc:686] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 1 LEADER]: Becoming Leader. State: Replica: d1f77df9a0d34ea68b9766d28e097414, State: Running, Role: LEADER
I20260926 07:02:35.243088 12588 consensus_queue.cc:237] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } }
I20260926 07:02:35.243767 12107 catalog_manager.cc:5495] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1f77df9a0d34ea68b9766d28e097414 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "d1f77df9a0d34ea68b9766d28e097414" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1f77df9a0d34ea68b9766d28e097414" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 35123 } health_report { overall_health: HEALTHY } } }
I20260926 07:02:35.254489 12577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.252338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42968 (local address 127.1.217.66:35123)
0926 07:02:35.252398 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:02:35.252399 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:02:35.252406 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:02:35.252432 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:02:35.252433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:02:35.252448 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:02:35.252507 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:02:35.252610 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.253130 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.253840 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:02:35.254014 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:02:35.254091 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:02:35.254234 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:02:35.254319 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:02:35.254378 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:02:35.254410 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 07:02:35.254425 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:02:35.256179 12520 raft_consensus.cc:1261] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 1 FOLLOWER]: Refusing update from remote peer d1f77df9a0d34ea68b9766d28e097414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:35.256206 12252 raft_consensus.cc:1261] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 1 FOLLOWER]: Refusing update from remote peer d1f77df9a0d34ea68b9766d28e097414: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:02:35.256523 12590 consensus_queue.cc:1035] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92184c0545a74172901bdab88ae430ab" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:35.256636 12588 consensus_queue.cc:1035] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f32dd01e0d94673838da61bb0c9f038" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 33935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:02:35.259080 12594 mvcc.cc:205] Tried to move back new op lower bound from 7333503611926786048 to 7333503611876360192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503611926786048}]
I20260926 07:02:35.259271 12593 mvcc.cc:205] Tried to move back new op lower bound from 7333503611926786048 to 7333503611876360192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503611926786048}]
I20260926 07:02:35.260582 12596 mvcc.cc:205] Tried to move back new op lower bound from 7333503611926786048 to 7333503611876360192. Current Snapshot: MvccSnapshot[applied={T|T < 7333503611926786048}]
I20260926 07:02:35.261901 12106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:35.261998 12106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:02:35.262717 12106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8a6717138b14f089eaa2389c288b108: Sending DeleteTablet for 3 replicas of tablet 19150bed1bdc45a2ac04813fd6019028
I20260926 07:02:35.263015 12500 tablet_service.cc:1514] Processing DeleteTablet for tablet 19150bed1bdc45a2ac04813fd6019028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:35 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53468
I20260926 07:02:35.263056 12232 tablet_service.cc:1514] Processing DeleteTablet for tablet 19150bed1bdc45a2ac04813fd6019028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:35 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:45534
I20260926 07:02:35.263260 12604 tablet_replica.cc:330] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab: stopping tablet replica
I20260926 07:02:35.263259 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 12169
I20260926 07:02:35.263283 12366 tablet_service.cc:1514] Processing DeleteTablet for tablet 19150bed1bdc45a2ac04813fd6019028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:02:35 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:42966
I20260926 07:02:35.263350 12604 raft_consensus.cc:2229] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:35.263442 12604 raft_consensus.cc:2258] T 19150bed1bdc45a2ac04813fd6019028 P 92184c0545a74172901bdab88ae430ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:35.264312 12603 tablet_replica.cc:330] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: stopping tablet replica
I20260926 07:02:35.264411 12603 raft_consensus.cc:2229] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:02:35.264508 12603 raft_consensus.cc:2258] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:35.264947 12603 ts_tablet_manager.cc:1905] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:02:35.266726 12603 ts_tablet_manager.cc:1918] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:35.266781 12603 log.cc:1196] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/19150bed1bdc45a2ac04813fd6019028
I20260926 07:02:35.267076 12603 ts_tablet_manager.cc:1939] T 19150bed1bdc45a2ac04813fd6019028 P 4f32dd01e0d94673838da61bb0c9f038: Deleting consensus metadata
I20260926 07:02:35.267441 12091 catalog_manager.cc:4841] TS 4f32dd01e0d94673838da61bb0c9f038 (127.1.217.67:33935): tablet 19150bed1bdc45a2ac04813fd6019028 (table test-table [id=9c841f726e6c46a0bfd408da685dd456]) successfully deleted
I20260926 07:02:35.268890 12605 tablet_replica.cc:330] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: stopping tablet replica
I20260926 07:02:35.268993 12605 raft_consensus.cc:2229] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:02:35.269120 12605 raft_consensus.cc:2258] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:02:35.269515 12605 ts_tablet_manager.cc:1905] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:02:35.270061 12092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:46839 (error 108)
I20260926 07:02:35.270489 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.270373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:46839 (local address 127.0.0.1:45542)
0926 07:02:35.270418 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.270447 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 07:02:35.270597 12092 catalog_manager.cc:4584] TS 92184c0545a74172901bdab88ae430ab (127.1.217.65:46839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19150bed1bdc45a2ac04813fd6019028: Network error: Client connection negotiation failed: client connection to 127.1.217.65:46839: connect: Connection refused (error 111)
I20260926 07:02:35.270888 12605 ts_tablet_manager.cc:1918] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:02:35.270972 12605 log.cc:1196] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/19150bed1bdc45a2ac04813fd6019028
I20260926 07:02:35.271015 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 12303
I20260926 07:02:35.271271 12605 ts_tablet_manager.cc:1939] T 19150bed1bdc45a2ac04813fd6019028 P d1f77df9a0d34ea68b9766d28e097414: Deleting consensus metadata
I20260926 07:02:35.276535 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 12437
I20260926 07:02:35.276619 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:02:35.276515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:35123 (local address 127.0.0.1:42984)
0926 07:02:35.276564 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:02:35.276594 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.66:35123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:02:35.281243 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 12075
2026-09-26T07:02:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1859 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12609](info): set up 2 sockets
Sep 26 07:02:35 dist-test-slave-t1mc krb5kdc[12609](info): commencing operation
krb5kdc: starting...
W20260926 07:02:36.313745 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:02:36 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:02:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:02:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangeradmin_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangeradmin_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangerlookup_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangerlookup_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
I20260926 07:02:36.467941 1893 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktn8ZA6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/postgres -l logfile start
2026-09-26 07:02:38.919 UTC [12639] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:02:38.919 UTC [12639] LOG: listening on IPv4 address "127.1.217.84", port 60353
2026-09-26 07:02:38.923 UTC [12639] LOG: listening on Unix socket "/tmp/.s.PGSQL.60353"
2026-09-26 07:02:38.944 UTC [12642] LOG: database system was shut down at 2026-09-26 07:02:37 UTC
2026-09-26 07:02:38.949 UTC [12639] LOG: database system is ready to accept connections
I20260926 07:02:39.889626 1893 mini_postgres.cc:96] Postgres bound to 60353
2026-09-26 07:02:39.895 UTC [12650] FATAL: database "slave" does not exist
127.1.217.84:60353 - accepting connections
I20260926 07:02:39.896026 1893 mini_ranger.cc:161] Starting Ranger...
I20260926 07:02:39.910665 1893 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:02:40.403175 1893 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:02:40.403257 1893 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin
2026-09-26 07:02:40,679 [I] DB FLAVOR :POSTGRES
2026-09-26 07:02:40,679 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:02:40,679 [I] Checking connection..
2026-09-26 07:02:40,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:02:40,921 [I] Checking connection passed.
2026-09-26 07:02:40,922 [I] --------- Verifying version history table ---------
2026-09-26 07:02:40,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:02:41,163 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:02:41,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:02:41,397 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:02:41,397 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:02:41,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:02:41.632 UTC [12714] WARNING: there is no transaction in progress
2026-09-26 07:02:41,639 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:02:41,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:02:41,883 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:02:41,883 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:02:41,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:02:42,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:02:42,359 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t1mc.c.gcp-upstream.internal','N') ;"
2026-09-26 07:02:42,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:02:42,812 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:02:42,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:02:43,070 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:02:43,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:02:43,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:02:43,535 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:02:43,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:02:44.484 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.503 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.520 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.535 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.551 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.568 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.621 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.628 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.642 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.652 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.662 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.676 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.685 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.691 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.698 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:44.705 UTC [12822] WARNING: there is no transaction in progress
2026-09-26 07:02:45,331 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:02:45,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:02:45,559 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:02:45,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:02:45,805 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:02:45,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:02:46,064 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:02:46,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:02:46,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:02:46,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t1mc.c.gcp-upstream.internal';"
2026-09-26 07:02:46,812 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:02:46,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:02:47,043 [I] DB_PATCHES have already been applied
I20260926 07:02:47.049350 1893 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktn8ZA6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:02:47.049412 1893 mini_ranger.cc:193] Using host: 127.1.217.84
I20260926 07:02:47.052268 1893 mini_ranger.cc:227] Ranger admin URL: http://127.1.217.84:45549
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:02:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:02:48.150573 1893 mini_ranger.cc:160] Time spent starting Ranger: real 8.255s user 0.000s sys 0.009s
Sep 26, 2026 7:03:08 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:03:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:03:08 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:03:08 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:03:09 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:03:09.622542 1893 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangerkms_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/rangerkms_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.217.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.217.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:03:09 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406189, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:03:09.711447 1893 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:03:09.720595 1893 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:03:10.610968 1893 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:03:10.611053 1893 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms
I20260926 07:03:10.611073 1893 mini_ranger_kms.cc:227] Using postgres at 127.1.217.84:60353
2026-09-26 07:03:10,783 [I] DB FLAVOR :POSTGRES
2026-09-26 07:03:10,784 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:03:10,784 [I] Checking connection
2026-09-26 07:03:10,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:03:11,018 [I] connection success
2026-09-26 07:03:11,018 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:03:11,018 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:03:11,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:03:11,250 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:03:11,250 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:03:11,250 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:03:11,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:60353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:03:11,516 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:03:11.973040 1893 mini_ranger_kms.cc:310] Created kms service
I20260926 07:03:12.164515 1893 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:03:12.220374 1893 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:03:12 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406189, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:12.608011 1893 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:03:12.608122 1893 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:03:12.608142 1893 mini_ranger_kms.cc:240] Using host: 127.1.217.84
I20260926 07:03:12.611399 1893 mini_ranger_kms.cc:276] Ranger KMS PID: 13002
I20260926 07:03:12.611521 1893 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.217.84:43423
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:03:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 07:03:14 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790406194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:14.829613 1893 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.118s user 0.001s sys 0.155s
I20260926 07:03:14.829744 1893 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:03:14.829825 1893 mini_ranger_kms.cc:401] 127.1.217.84:43423/kms/v1/keys
Sep 26 07:03:15 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
Sep 26 07:03:15 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
Sep 26 07:03:17 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:03:17.592432 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37429
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:37429
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:43423/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:03:17.693549 13053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:03:17.693794 13053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:03:17.693828 13053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:03:17.694960 13053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:03:17.695029 13053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:03:17.695051 13053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:03:17.695073 13053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:03:17.695103 13053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:03:17.696617 13053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.217.84:43423/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:37429
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:03:17.696904 13053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:03:17.697124 13053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:03:17.702343 13061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:03:17.702353 13058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:17.702483 13053 server_base.cc:1034] running on GCE node
W20260926 07:03:17.702353 13059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:17.702855 13053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:03:17.703188 13053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:03:17.704356 13053 hybrid_clock.cc:648] HybridClock initialized: now 1790406197704332 us; error 41 us; skew 500 ppm
Sep 26 07:03:17 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:17.711361 13053 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:03:17.711911 13053 webserver.cc:460] Webserver started at http://127.1.217.126:41297/ using document root <none> and password file <none>
I20260926 07:03:17.712141 13053 fs_manager.cc:359] Metadata directory not provided
I20260926 07:03:17.712201 13053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:03:17.712373 13053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:03:17 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406197, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:17.772385 13053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "e0114cccf7ea4dfb8f70a9ded7704af1"
format_stamp: "Formatted at 2026-09-26 07:03:17 on dist-test-slave-t1mc"
server_key: "23301515c3f3c7e1f2296bf084a90edb"
server_key_iv: "de91f912fd43a4e681a24d8f18077423"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:17.772887 13053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "e0114cccf7ea4dfb8f70a9ded7704af1"
format_stamp: "Formatted at 2026-09-26 07:03:17 on dist-test-slave-t1mc"
server_key: "23301515c3f3c7e1f2296bf084a90edb"
server_key_iv: "de91f912fd43a4e681a24d8f18077423"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:17.774606 13053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:03:17.788833 13068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:17.789083 13053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:03:17.789186 13053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "e0114cccf7ea4dfb8f70a9ded7704af1"
format_stamp: "Formatted at 2026-09-26 07:03:17 on dist-test-slave-t1mc"
server_key: "23301515c3f3c7e1f2296bf084a90edb"
server_key_iv: "de91f912fd43a4e681a24d8f18077423"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:17.789287 13053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:03:17.810817 13053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:03:17.812844 13053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:03:17.812995 13053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:03:17.817785 13053 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:37429
I20260926 07:03:17.817829 13120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:37429 every 8 connection(s)
I20260926 07:03:17.818182 13053 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:03:17.818943 13121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:03:17.819566 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 13053
I20260926 07:03:17.819710 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:03:17.819931 1893 external_mini_cluster.cc:1428] Setting key 091a3f3fe9d9edcbd80341daae8324f1
I20260926 07:03:17.822032 13121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: Bootstrap starting.
I20260926 07:03:17.822708 13121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: Neither blocks nor log segments found. Creating new log.
I20260926 07:03:17.823004 13121 log.cc:824] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:03:17.823797 13121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: No bootstrap required, opened a new log
I20260926 07:03:17.826749 13121 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } }
I20260926 07:03:17.826933 13121 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:03:17.826977 13121 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0114cccf7ea4dfb8f70a9ded7704af1, State: Initialized, Role: FOLLOWER
I20260926 07:03:17.827157 13121 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } }
I20260926 07:03:17.827257 13121 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:03:17.827296 13121 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:03:17.827373 13121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 07:03:17 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:03:17.828078 13121 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } }
I20260926 07:03:17.828228 13121 leader_election.cc:302] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0114cccf7ea4dfb8f70a9ded7704af1; no voters:
I20260926 07:03:17.828414 13121 leader_election.cc:288] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:03:17.828445 13126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:03:17.828651 13126 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [term 1 LEADER]: Becoming Leader. State: Replica: e0114cccf7ea4dfb8f70a9ded7704af1, State: Running, Role: LEADER
I20260926 07:03:17.828787 13121 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:03:17.828805 13126 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } }
I20260926 07:03:17.829324 13126 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0114cccf7ea4dfb8f70a9ded7704af1. Latest consensus state: current_term: 1 leader_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } } }
I20260926 07:03:17.829469 13126 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:03:17.829675 13128 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0114cccf7ea4dfb8f70a9ded7704af1" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37429 } } }
I20260926 07:03:17.829823 13128 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1 [sys.catalog]: This master's current role is: LEADER
I20260926 07:03:17.830011 13130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:03:17.831152 13130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:03:17.831416 13121 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:03:17.833961 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:17.820464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60810 (local address 127.1.217.126:37429)
0926 07:03:17.820694 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:17.820702 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:17.820724 (+ 22us) server_negotiation.cc:407] Connection header received
0926 07:03:17.821166 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:17.821172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:17.821264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:17.821413 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:03:17.822041 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:17.823054 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:17.824107 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:17.824728 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:17.824898 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:17.828168 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:03:17.828172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:03:17.828173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:03:17.828184 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:03:17.830233 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:17.830669 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:17.830673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:17.830673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:17.830786 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:17.831032 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:17.831033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:17.831033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:17.831328 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:03:17.831864 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:17.831945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 07:03:17.832070 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":95,"threads_started":1}
I20260926 07:03:17.835820 13130 catalog_manager.cc:1347] Generated new cluster ID: 4cccbc286f6e44eab72e0ffae5378b24
I20260926 07:03:17.835892 13130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:03:17.864096 13130 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:03:17.864945 13130 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:03:17.875617 13130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: Generated new TSK 0
I20260926 07:03:17.875918 13130 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:03:18.595852 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:43423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:03:18.683553 13166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:03:18.683770 13166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:03:18.683806 13166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:03:18.685173 13166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:03:18.685241 13166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:03:18.685286 13166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:03:18.686738 13166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.217.84:43423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:03:18.686980 13166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:03:18.687203 13166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:03:18.692550 13174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:03:18.692538 13172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:03:18.692550 13171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:18.693044 13166 server_base.cc:1034] running on GCE node
I20260926 07:03:18.693249 13166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:03:18.693504 13166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:03:18.694653 13166 hybrid_clock.cc:648] HybridClock initialized: now 1790406198694637 us; error 36 us; skew 500 ppm
Sep 26 07:03:18 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:18.697037 13166 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:03:18.697496 13166 webserver.cc:460] Webserver started at http://127.1.217.65:42437/ using document root <none> and password file <none>
I20260926 07:03:18.697733 13166 fs_manager.cc:359] Metadata directory not provided
I20260926 07:03:18.697777 13166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:03:18.697906 13166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:03:18 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:18.710690 13166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "c8b510218cb94bbb8ee5600d357babe5"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "a3c52361e823ca107b6e85e7961c3288"
server_key_iv: "9da2c93f6cc430f763503b8ee907ad2d"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.711150 13166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "c8b510218cb94bbb8ee5600d357babe5"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "a3c52361e823ca107b6e85e7961c3288"
server_key_iv: "9da2c93f6cc430f763503b8ee907ad2d"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.714066 13166 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 07:03:18.727846 13181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:18.728147 13166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:03:18.728253 13166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "c8b510218cb94bbb8ee5600d357babe5"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "a3c52361e823ca107b6e85e7961c3288"
server_key_iv: "9da2c93f6cc430f763503b8ee907ad2d"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.728361 13166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:03:18.747030 13166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:03:18.749547 13166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:03:18.749713 13166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:03:18.749953 13166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:03:18.750301 13166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:03:18.750408 13166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:18.750514 13166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:03:18.750597 13166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:18.758136 13166 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:40305
I20260926 07:03:18.758661 13166 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:03:18.761471 13294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:40305 every 8 connection(s)
I20260926 07:03:18.761610 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 13166
I20260926 07:03:18.761732 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:03:18.761961 1893 external_mini_cluster.cc:1428] Setting key 89ef094bc209e03a5144afcdbc3618a2
Sep 26 07:03:18 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:03:18.778177 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:18.761414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:37507 (local address 127.1.217.126:37429)
0926 07:03:18.767644 (+ 6230us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:18.767646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:18.767653 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:03:18.767674 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:18.767674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:18.767686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:18.767766 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:03:18.768379 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:18.769038 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:18.771581 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:18.771896 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:18.772000 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:18.774744 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:03:18.774745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:03:18.774745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:03:18.774753 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:03:18.776610 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:18.777109 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:18.777111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:18.777111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:18.777192 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:18.777466 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:18.777467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:18.777468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:18.777595 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:03:18.777671 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:18.777960 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 07:03:18.778048 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6184,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
I20260926 07:03:18.778496 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:18.761106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37429 (local address 127.1.217.65:37507)
0926 07:03:18.764277 (+ 3171us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:18.764329 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:18.767459 (+ 3130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:18.767869 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:18.767876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:18.768072 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:03:18.768275 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:18.768280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:18.769164 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:18.769165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:18.770548 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:18.770551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:18.772012 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:18.772013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:18.772169 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:18.772837 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:03:18.772848 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:03:18.774617 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:03:18.776719 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:18.776726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:18.776728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:18.777039 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:18.777266 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:18.777267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:18.777267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:18.777386 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:18.777665 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:03:18.777671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:03:18.777817 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 07:03:18.777926 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:03:18.778078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3117,"thread_start_us":89,"threads_started":1}
I20260926 07:03:18.779165 13295 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37429
I20260926 07:03:18.779266 13295 heartbeater.cc:464] Registering TS with master...
I20260926 07:03:18.779460 13295 heartbeater.cc:511] Master 127.1.217.126:37429 requested a full tablet report, sending...
I20260926 07:03:18.779958 13085 ts_manager.cc:194] Registered new tserver with Master: c8b510218cb94bbb8ee5600d357babe5 (127.1.217.65:40305)
I20260926 07:03:18.780716 13085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:37507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:03:18.804164 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:43423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:03:18.924335 13303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:03:18.924677 13303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:03:18.924755 13303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:03:18.926472 13303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:03:18.926575 13303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:03:18.926653 13303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:03:18.928825 13303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.217.84:43423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:03:18.929370 13303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:03:18.929669 13303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:03:18.936124 13313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:03:18.936156 13311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:18.937498 13303 server_base.cc:1034] running on GCE node
W20260926 07:03:18.938894 13310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:18.939191 13303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:03:18.939466 13303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:03:18.940625 13303 hybrid_clock.cc:648] HybridClock initialized: now 1790406198940599 us; error 41 us; skew 500 ppm
Sep 26 07:03:18 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:18.943915 13303 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:03:18.944406 13303 webserver.cc:460] Webserver started at http://127.1.217.66:40807/ using document root <none> and password file <none>
I20260926 07:03:18.944644 13303 fs_manager.cc:359] Metadata directory not provided
I20260926 07:03:18.944700 13303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:03:18.944869 13303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:03:18 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:18.959470 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "eff54df7c3c7437fb2ac3ef271576296"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "abeeb0cab486debee4332e7b75f7ba46"
server_key_iv: "e788b8e532ce7cef52102ed955742578"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.960026 13303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "eff54df7c3c7437fb2ac3ef271576296"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "abeeb0cab486debee4332e7b75f7ba46"
server_key_iv: "e788b8e532ce7cef52102ed955742578"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.965152 13303 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 07:03:18.986816 13320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:18.987090 13303 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260926 07:03:18.987178 13303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "eff54df7c3c7437fb2ac3ef271576296"
format_stamp: "Formatted at 2026-09-26 07:03:18 on dist-test-slave-t1mc"
server_key: "abeeb0cab486debee4332e7b75f7ba46"
server_key_iv: "e788b8e532ce7cef52102ed955742578"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:18.987272 13303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:19.006162 13303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:03:19.008567 13303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:03:19.008797 13303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:03:19.009091 13303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:03:19.009557 13303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:03:19.009804 13303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:19.009877 13303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:03:19.009936 13303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:19.016747 13303 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:37643
I20260926 07:03:19.019773 13303 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:03:19.019222 13433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:37643 every 8 connection(s)
I20260926 07:03:19.024202 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 13303
I20260926 07:03:19.024307 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:03:19.024482 1893 external_mini_cluster.cc:1428] Setting key 81c49ae09eacf494ce1904515fdd906c
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406198, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:03:19.034313 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.021302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:55771 (local address 127.1.217.126:37429)
0926 07:03:19.021367 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.021369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.021509 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:03:19.022166 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.022168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.022184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.022227 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:03:19.023022 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.023791 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.027999 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.028318 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.028361 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.030975 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:03:19.030977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:03:19.030977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:03:19.030986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:03:19.032683 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.033232 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.033233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.033234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.033298 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.033570 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.033570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.033571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.033706 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:03:19.033755 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.034112 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.034224 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:03:19.035127 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.021012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37429 (local address 127.1.217.66:55771)
0926 07:03:19.021386 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.021430 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.022044 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.022415 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.022420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.022653 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:03:19.022915 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.022921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.023907 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.023908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.027888 (+ 3980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.027892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.028452 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.028453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.028590 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.029240 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:03:19.029251 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:03:19.030863 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:03:19.032822 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:19.032830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:19.032833 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:19.033132 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:19.033368 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:19.033368 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:19.033368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:19.033502 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:19.033787 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:03:19.033794 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:03:19.033965 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.034706 (+ 741us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.034889 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":83,"threads_started":1}
I20260926 07:03:19.035795 13434 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37429
I20260926 07:03:19.035887 13434 heartbeater.cc:464] Registering TS with master...
I20260926 07:03:19.036087 13434 heartbeater.cc:511] Master 127.1.217.126:37429 requested a full tablet report, sending...
I20260926 07:03:19.036500 13085 ts_manager.cc:194] Registered new tserver with Master: eff54df7c3c7437fb2ac3ef271576296 (127.1.217.66:37643)
I20260926 07:03:19.037057 13085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:55771
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:03:19.069785 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:43423/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:03:19.192978 13439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:03:19.193328 13439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:03:19.193392 13439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:03:19.195261 13439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:03:19.195349 13439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:03:19.195415 13439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:03:19.197633 13439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.217.84:43423/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37429
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:03:19.198022 13439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:03:19.198369 13439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:03:19.205122 13446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:03:19.205463 13445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:19.205725 13439 server_base.cc:1034] running on GCE node
W20260926 07:03:19.206297 13448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:03:19.206578 13439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:03:19.206852 13439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:03:19.208014 13439 hybrid_clock.cc:648] HybridClock initialized: now 1790406199207992 us; error 33 us; skew 500 ppm
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:03:19.211364 13439 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:03:19.211971 13439 webserver.cc:460] Webserver started at http://127.1.217.67:39101/ using document root <none> and password file <none>
I20260926 07:03:19.212240 13439 fs_manager.cc:359] Metadata directory not provided
I20260926 07:03:19.212291 13439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:03:19.212414 13439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:19.231914 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "c5a7ad848e774f41b9d58bd1853df4e0"
format_stamp: "Formatted at 2026-09-26 07:03:19 on dist-test-slave-t1mc"
server_key: "02ee18758baa6f0b127acc38f2f9e8bf"
server_key_iv: "f82695ced0990cb9af4b307249653579"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:19.232434 13439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "c5a7ad848e774f41b9d58bd1853df4e0"
format_stamp: "Formatted at 2026-09-26 07:03:19 on dist-test-slave-t1mc"
server_key: "02ee18758baa6f0b127acc38f2f9e8bf"
server_key_iv: "f82695ced0990cb9af4b307249653579"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:19.238420 13439 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260926 07:03:19.253373 13455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:19.253648 13439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:03:19.253751 13439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "c5a7ad848e774f41b9d58bd1853df4e0"
format_stamp: "Formatted at 2026-09-26 07:03:19 on dist-test-slave-t1mc"
server_key: "02ee18758baa6f0b127acc38f2f9e8bf"
server_key_iv: "f82695ced0990cb9af4b307249653579"
server_key_version: "kuduclusterkey@0"
I20260926 07:03:19.253860 13439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:03:19.301579 13439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:03:19.304281 13439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:03:19.304452 13439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:03:19.304741 13439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:03:19.305184 13439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:03:19.305241 13439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:19.305279 13439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:03:19.305308 13439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:03:19.314576 13439 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:40985
I20260926 07:03:19.315039 13439 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:03:19.316016 13568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:40985 every 8 connection(s)
I20260926 07:03:19.320326 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 13439
I20260926 07:03:19.320432 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:03:19.320624 1893 external_mini_cluster.cc:1428] Setting key 28c4325fa18045213850e612d8d3c295
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:03:19.331993 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.320108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:48775 (local address 127.1.217.126:37429)
0926 07:03:19.320799 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.320802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.320813 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:03:19.320841 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.320842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.320853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.320928 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:03:19.323101 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.323946 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.325555 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.325870 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.325957 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.328487 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:03:19.328489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:03:19.328489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:03:19.328498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:03:19.330294 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.330843 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.330844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.330845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.330903 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.331218 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.331219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.331220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.331372 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:03:19.331436 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.331807 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.331891 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260926 07:03:19.332821 13571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.316716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37429 (local address 127.1.217.67:48775)
0926 07:03:19.316911 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.316961 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.317545 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.322425 (+ 4880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.322431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.322681 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:03:19.322982 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.322989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.324071 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.324072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.325449 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.325453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.325977 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.325978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.326127 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.326757 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:03:19.326768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:03:19.328378 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:03:19.330412 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:19.330418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:19.330421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:19.330738 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:19.330974 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:03:19.330975 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:03:19.330975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:03:19.331124 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:03:19.331464 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:03:19.331470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:03:19.331632 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.332381 (+ 749us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.332555 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":132,"threads_started":1}
I20260926 07:03:19.333295 13569 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37429
I20260926 07:03:19.333407 13569 heartbeater.cc:464] Registering TS with master...
I20260926 07:03:19.333650 13569 heartbeater.cc:511] Master 127.1.217.126:37429 requested a full tablet report, sending...
I20260926 07:03:19.334106 13085 ts_manager.cc:194] Registered new tserver with Master: c5a7ad848e774f41b9d58bd1853df4e0 (127.1.217.67:40985)
I20260926 07:03:19.334642 13085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:48775
I20260926 07:03:19.344874 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:03:19.354408 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.346535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60818 (local address 127.1.217.126:37429)
0926 07:03:19.346579 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.346581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.346719 (+ 138us) server_negotiation.cc:407] Connection header received
0926 07:03:19.346772 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.346772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.346784 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.346845 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:03:19.347566 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.348344 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.349461 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.349771 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.349849 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.351342 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:03:19.351343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:03:19.351344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:03:19.351351 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:03:19.353146 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.353546 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.353547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.353548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.353612 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:03:19.353844 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:03:19.353845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:03:19.353846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:03:19.353990 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:03:19.354095 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.354231 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.354311 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:03:19.356277 13085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:03:19.358559 13085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:03:19.367776 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.361616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:37643 (local address 127.0.0.1:41590)
0926 07:03:19.362462 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.362473 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.362530 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.363063 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.363066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.363069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:03:19.363134 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.363136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.364729 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.364730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.366090 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.366092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.367477 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.367478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.367558 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.367571 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.367642 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.367694 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":108,"threads_started":1}
I20260926 07:03:19.368191 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.362129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.1.217.66:37643)
0926 07:03:19.362898 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.362902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.362911 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:03:19.362946 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.362948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.362962 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.363039 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:03:19.363229 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.364632 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.366216 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.367292 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.367364 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.367443 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.368069 (+ 626us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.368110 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":76,"threads_started":1}
I20260926 07:03:19.369040 13368 tablet_service.cc:1467] Processing CreateTablet for tablet 46f78df0218240b88a8604b0999171b3 (DEFAULT_TABLE table=test-table [id=e92931bab3064750917cb02b784815f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:03:19.369446 13368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f78df0218240b88a8604b0999171b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:03:19.370528 13584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.361623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:40985 (local address 127.0.0.1:40488)
0926 07:03:19.362416 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.362456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.362531 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.363934 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.363934 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.363936 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:03:19.363975 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.363976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.366495 (+ 2519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.366496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.367540 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.367543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.370288 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.370289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.370358 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.370362 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.370414 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.370451 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":80,"threads_started":1}
I20260926 07:03:19.372228 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.361881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40488 (local address 127.1.217.67:40985)
0926 07:03:19.362090 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.362096 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.362620 (+ 524us) server_negotiation.cc:407] Connection header received
0926 07:03:19.362677 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.362679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.362696 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.362764 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:03:19.364046 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.365206 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.368574 (+ 3368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.369359 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.369430 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.369502 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.372104 (+ 2602us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.372146 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260926 07:03:19.371935 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.361852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:40305 (local address 127.0.0.1:59384)
0926 07:03:19.362660 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.362668 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.362703 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.363353 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.363354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.363356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:03:19.363855 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.363857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.367647 (+ 3790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.367648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.370137 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.370141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.371699 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.371700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.371774 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.371778 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.371831 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.371867 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":70,"threads_started":1}
I20260926 07:03:19.373229 13503 tablet_service.cc:1467] Processing CreateTablet for tablet 46f78df0218240b88a8604b0999171b3 (DEFAULT_TABLE table=test-table [id=e92931bab3064750917cb02b784815f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:03:19.373387 13588 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Bootstrap starting.
I20260926 07:03:19.373555 13503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f78df0218240b88a8604b0999171b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:03:19.374385 13588 tablet_bootstrap.cc:654] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Neither blocks nor log segments found. Creating new log.
I20260926 07:03:19.374686 13588 log.cc:824] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Log is configured to *not* fsync() on all Append() calls
I20260926 07:03:19.375423 13588 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: No bootstrap required, opened a new log
I20260926 07:03:19.375486 13588 ts_tablet_manager.cc:1397] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:03:19.377249 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.362283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.1.217.65:40305)
0926 07:03:19.363152 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.363156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.363165 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:03:19.363195 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.363197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.363211 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.363285 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:03:19.365309 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.366396 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.370797 (+ 4401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.371520 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.371594 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.371663 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.377117 (+ 5454us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.377177 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":49,"threads_started":1}
I20260926 07:03:19.377749 13588 raft_consensus.cc:348] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.377916 13588 raft_consensus.cc:374] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:03:19.377954 13588 raft_consensus.cc:729] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff54df7c3c7437fb2ac3ef271576296, State: Initialized, Role: FOLLOWER
I20260926 07:03:19.378084 13588 consensus_queue.cc:260] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.378000 13229 tablet_service.cc:1467] Processing CreateTablet for tablet 46f78df0218240b88a8604b0999171b3 (DEFAULT_TABLE table=test-table [id=e92931bab3064750917cb02b784815f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:03:19.378343 13229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f78df0218240b88a8604b0999171b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:03:19.378356 13588 ts_tablet_manager.cc:1428] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:03:19.378669 13434 heartbeater.cc:503] Master 127.1.217.126:37429 was elected leader, sending a full tablet report...
I20260926 07:03:19.380074 13590 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Bootstrap starting.
I20260926 07:03:19.381479 13590 tablet_bootstrap.cc:654] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Neither blocks nor log segments found. Creating new log.
I20260926 07:03:19.381820 13592 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Bootstrap starting.
I20260926 07:03:19.381839 13590 log.cc:824] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:03:19.382560 13590 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: No bootstrap required, opened a new log
I20260926 07:03:19.382625 13590 ts_tablet_manager.cc:1397] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:03:19.382974 13592 tablet_bootstrap.cc:654] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Neither blocks nor log segments found. Creating new log.
I20260926 07:03:19.383244 13592 log.cc:824] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Log is configured to *not* fsync() on all Append() calls
I20260926 07:03:19.384546 13590 raft_consensus.cc:348] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.384699 13590 raft_consensus.cc:374] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:03:19.384729 13590 raft_consensus.cc:729] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5a7ad848e774f41b9d58bd1853df4e0, State: Initialized, Role: FOLLOWER
I20260926 07:03:19.384864 13590 consensus_queue.cc:260] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.385120 13590 ts_tablet_manager.cc:1428] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:03:19.385234 13569 heartbeater.cc:503] Master 127.1.217.126:37429 was elected leader, sending a full tablet report...
I20260926 07:03:19.385584 13592 tablet_bootstrap.cc:492] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: No bootstrap required, opened a new log
I20260926 07:03:19.385653 13592 ts_tablet_manager.cc:1397] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 07:03:19.388103 13592 raft_consensus.cc:348] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.388267 13592 raft_consensus.cc:374] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:03:19.388309 13592 raft_consensus.cc:729] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b510218cb94bbb8ee5600d357babe5, State: Initialized, Role: FOLLOWER
I20260926 07:03:19.388433 13592 consensus_queue.cc:260] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.388693 13592 ts_tablet_manager.cc:1428] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:03:19.388711 13295 heartbeater.cc:503] Master 127.1.217.126:37429 was elected leader, sending a full tablet report...
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
W20260926 07:03:19.510882 13296 tablet.cc:2369] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:03:19.520879 13435 tablet.cc:2369] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:03:19.566735 13570 tablet.cc:2369] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:03:19.637436 13591 raft_consensus.cc:482] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:03:19.637595 13591 raft_consensus.cc:504] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.637972 13591 leader_election.cc:288] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8b510218cb94bbb8ee5600d357babe5 (127.1.217.65:40305), c5a7ad848e774f41b9d58bd1853df4e0 (127.1.217.67:40985)
I20260926 07:03:19.641861 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.638187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:40305 (local address 127.1.217.66:53935)
0926 07:03:19.638371 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.638383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.638440 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.638709 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.638710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.638713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:03:19.638780 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.638782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.639671 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.639672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.640855 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.640858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.641623 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.641624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.641693 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.641697 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.641739 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.641777 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260926 07:03:19.641901 13585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.638413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:32879 (local address 127.1.217.67:40985)
0926 07:03:19.638454 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.638457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.638572 (+ 115us) server_negotiation.cc:407] Connection header received
0926 07:03:19.638683 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.638684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.638696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.638726 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:03:19.638930 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.639632 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.640993 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.641512 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.641561 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.641569 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.641793 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.641827 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:03:19.642122 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.638256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:40985 (local address 127.1.217.66:32879)
0926 07:03:19.638522 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.638530 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:03:19.638595 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:03:19.638792 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:03:19.638792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:03:19.638794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:03:19.638862 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.638864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.639769 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.639769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.640895 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:03:19.640898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.641626 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.641627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:03:19.641718 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.641721 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:03:19.642005 (+ 284us) client_negotiation.cc:240] Negotiation successful
0926 07:03:19.642046 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":1,"thread_start_us":58,"threads_started":1}
I20260926 07:03:19.642289 13587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.638496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:53935 (local address 127.1.217.65:40305)
0926 07:03:19.638534 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.638536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.638544 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:03:19.638584 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.638585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.638593 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.638659 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:03:19.638846 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.639580 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.640964 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.641505 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.641549 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.641567 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.642186 (+ 619us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.642227 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:03:19.642517 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f78df0218240b88a8604b0999171b3" candidate_uuid: "eff54df7c3c7437fb2ac3ef271576296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b510218cb94bbb8ee5600d357babe5" is_pre_election: true
I20260926 07:03:19.642693 13247 raft_consensus.cc:2454] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff54df7c3c7437fb2ac3ef271576296 in term 0.
I20260926 07:03:19.642745 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f78df0218240b88a8604b0999171b3" candidate_uuid: "eff54df7c3c7437fb2ac3ef271576296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" is_pre_election: true
I20260926 07:03:19.642890 13523 raft_consensus.cc:2454] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff54df7c3c7437fb2ac3ef271576296 in term 0.
I20260926 07:03:19.642938 13321 leader_election.cc:302] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8b510218cb94bbb8ee5600d357babe5, eff54df7c3c7437fb2ac3ef271576296; no voters:
I20260926 07:03:19.643150 13591 raft_consensus.cc:2781] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:03:19.643214 13591 raft_consensus.cc:482] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:03:19.643234 13591 raft_consensus.cc:3037] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:03:19.644575 13591 raft_consensus.cc:504] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.644701 13591 leader_election.cc:288] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [CANDIDATE]: Term 1 election: Requested vote from peers c8b510218cb94bbb8ee5600d357babe5 (127.1.217.65:40305), c5a7ad848e774f41b9d58bd1853df4e0 (127.1.217.67:40985)
I20260926 07:03:19.644925 13523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f78df0218240b88a8604b0999171b3" candidate_uuid: "eff54df7c3c7437fb2ac3ef271576296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5a7ad848e774f41b9d58bd1853df4e0"
I20260926 07:03:19.645006 13523 raft_consensus.cc:3037] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:03:19.645813 13523 raft_consensus.cc:2454] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff54df7c3c7437fb2ac3ef271576296 in term 1.
I20260926 07:03:19.646014 13324 leader_election.cc:302] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5a7ad848e774f41b9d58bd1853df4e0, eff54df7c3c7437fb2ac3ef271576296; no voters:
I20260926 07:03:19.646095 13591 raft_consensus.cc:2781] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:03:19.646212 13591 raft_consensus.cc:686] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 1 LEADER]: Becoming Leader. State: Replica: eff54df7c3c7437fb2ac3ef271576296, State: Running, Role: LEADER
I20260926 07:03:19.646288 13591 consensus_queue.cc:237] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } }
I20260926 07:03:19.646771 13247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f78df0218240b88a8604b0999171b3" candidate_uuid: "eff54df7c3c7437fb2ac3ef271576296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8b510218cb94bbb8ee5600d357babe5"
I20260926 07:03:19.646852 13247 raft_consensus.cc:3037] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:03:19.647404 13085 catalog_manager.cc:5495] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff54df7c3c7437fb2ac3ef271576296 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "eff54df7c3c7437fb2ac3ef271576296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff54df7c3c7437fb2ac3ef271576296" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 37643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 } health_report { overall_health: UNKNOWN } } }
I20260926 07:03:19.647742 13247 raft_consensus.cc:2454] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff54df7c3c7437fb2ac3ef271576296 in term 1.
Sep 26 07:03:19 dist-test-slave-t1mc krb5kdc[12609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406199, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:19.722636 13586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.719048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.1.217.66:37643)
0926 07:03:19.719135 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 07:03:19.719137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:03:19.719145 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:03:19.719187 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:03:19.719189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:03:19.719212 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:03:19.719280 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:03:19.719615 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.720342 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.721524 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:03:19.721860 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:03:19.721926 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:03:19.722136 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:03:19.722272 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:03:19.722348 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:03:19.722514 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 07:03:19.722533 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 07:03:19.728293 13523 raft_consensus.cc:1261] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 1 FOLLOWER]: Refusing update from remote peer eff54df7c3c7437fb2ac3ef271576296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:03:19.728668 13599 consensus_queue.cc:1035] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5a7ad848e774f41b9d58bd1853df4e0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:03:19.728878 13247 raft_consensus.cc:1261] T 46f78df0218240b88a8604b0999171b3 P c8b510218cb94bbb8ee5600d357babe5 [term 1 FOLLOWER]: Refusing update from remote peer eff54df7c3c7437fb2ac3ef271576296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:03:19.730652 13599 consensus_queue.cc:1035] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8b510218cb94bbb8ee5600d357babe5" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 40305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:03:19.734114 13603 mvcc.cc:205] Tried to move back new op lower bound from 7333503794082504704 to 7333503793751724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333503794082504704}]
I20260926 07:03:19.734743 13606 mvcc.cc:205] Tried to move back new op lower bound from 7333503794082504704 to 7333503793751724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333503794082504704}]
I20260926 07:03:19.744818 13085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:03:19.744935 13085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60818:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:03:19.745663 13085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0114cccf7ea4dfb8f70a9ded7704af1: Sending DeleteTablet for 3 replicas of tablet 46f78df0218240b88a8604b0999171b3
I20260926 07:03:19.746021 13503 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f78df0218240b88a8604b0999171b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:03:19 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:40488
I20260926 07:03:19.746095 13368 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f78df0218240b88a8604b0999171b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:03:19 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:41590
I20260926 07:03:19.746309 13229 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f78df0218240b88a8604b0999171b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:03:19 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:59384
I20260926 07:03:19.747056 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 13166
I20260926 07:03:19.747112 13604 mvcc.cc:205] Tried to move back new op lower bound from 7333503794082504704 to 7333503793751724032. Current Snapshot: MvccSnapshot[applied={T|T < 7333503794082504704 or (T in {7333503794082504704})}]
I20260926 07:03:19.747251 13613 tablet_replica.cc:330] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296: stopping tablet replica
I20260926 07:03:19.747339 13613 raft_consensus.cc:2229] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:03:19.747417 13613 raft_consensus.cc:2258] T 46f78df0218240b88a8604b0999171b3 P eff54df7c3c7437fb2ac3ef271576296 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 07:03:19.754248 13069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:40305 (error 108)
I20260926 07:03:19.755317 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 13303
I20260926 07:03:19.759356 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.759223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:40305 (local address 127.0.0.1:59396)
0926 07:03:19.759277 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.759317 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:40305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:03:19.759609 13069 catalog_manager.cc:4584] TS c8b510218cb94bbb8ee5600d357babe5 (127.1.217.65:40305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46f78df0218240b88a8604b0999171b3: Network error: Client connection negotiation failed: client connection to 127.1.217.65:40305: connect: Connection refused (error 111)
I20260926 07:03:19.761097 13614 tablet_replica.cc:330] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: stopping tablet replica
I20260926 07:03:19.761497 13614 raft_consensus.cc:2229] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:03:19.761998 13614 raft_consensus.cc:2258] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:03:19.763403 13614 ts_tablet_manager.cc:1905] T 46f78df0218240b88a8604b0999171b3 P c5a7ad848e774f41b9d58bd1853df4e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:03:19.764151 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.764047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:37643 (local address 127.0.0.1:41618)
0926 07:03:19.764089 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.764112 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.66:37643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 07:03:19.764305 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 13439
I20260926 07:03:19.773339 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:03:19.772713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:40985 (local address 127.0.0.1:40490)
0926 07:03:19.772758 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:03:19.773297 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.67:40985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 07:03:19.776650 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 13053
I20260926 07:03:19.786355 1893 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:03:20.130559 1893 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:03:20.130712 1893 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:03:20.474834 1893 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:03:20.475 UTC [12639] LOG: received smart shutdown request
2026-09-26 07:03:20.478 UTC [12639] LOG: background worker "logical replication launcher" (PID 12648) exited with exit code 1
2026-09-26 07:03:20.479 UTC [12643] LOG: shutting down
2026-09-26 07:03:20.656 UTC [12639] LOG: database system is shut down
2026-09-26T07:03:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45489 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:03:20 dist-test-slave-t1mc krb5kdc[13626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:03:20 dist-test-slave-t1mc krb5kdc[13626](info): set up 2 sockets
Sep 26 07:03:20 dist-test-slave-t1mc krb5kdc[13626](info): commencing operation
krb5kdc: starting...
W20260926 07:03:21.804716 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:03:21 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:03:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangeradmin_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangeradmin_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangerlookup_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangerlookup_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
I20260926 07:03:21.964751 1893 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktn8ZA6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/postgres -l logfile start
2026-09-26 07:03:24.525 UTC [13656] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 07:03:24.525 UTC [13656] LOG: listening on IPv4 address "127.1.217.84", port 47619
2026-09-26 07:03:24.529 UTC [13656] LOG: listening on Unix socket "/tmp/.s.PGSQL.47619"
2026-09-26 07:03:24.550 UTC [13659] LOG: database system was shut down at 2026-09-26 07:03:23 UTC
2026-09-26 07:03:24.554 UTC [13656] LOG: database system is ready to accept connections
I20260926 07:03:25.492543 1893 mini_postgres.cc:96] Postgres bound to 47619
2026-09-26 07:03:25.498 UTC [13667] FATAL: database "slave" does not exist
127.1.217.84:47619 - accepting connections
I20260926 07:03:25.498911 1893 mini_ranger.cc:161] Starting Ranger...
I20260926 07:03:25.513116 1893 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 07:03:25.991706 1893 mini_ranger.cc:87] Created ranger Postgres database
I20260926 07:03:25.991789 1893 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin
2026-09-26 07:03:26,257 [I] DB FLAVOR :POSTGRES
2026-09-26 07:03:26,257 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:03:26,258 [I] Checking connection..
2026-09-26 07:03:26,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 07:03:26,503 [I] Checking connection passed.
2026-09-26 07:03:26,503 [I] --------- Verifying version history table ---------
2026-09-26 07:03:26,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:03:26,764 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:03:26,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:03:26,991 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 07:03:26,992 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 07:03:26,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 07:03:27.246 UTC [13731] WARNING: there is no transaction in progress
2026-09-26 07:03:27,252 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 07:03:27,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 07:03:27,481 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 07:03:27,482 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:03:27,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 07:03:27,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 07:03:27,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t1mc.c.gcp-upstream.internal','N') ;"
2026-09-26 07:03:28,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:03:28,403 [I] Table x_portal_user does not exist in database ranger
2026-09-26 07:03:28,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:03:28,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 07:03:28,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:03:28,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:03:29,157 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 07:03:29,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 07:03:30.122 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.138 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.154 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.170 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.185 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.206 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.261 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.267 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.282 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.292 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.302 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.317 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.326 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.333 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.340 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30.347 UTC [13839] WARNING: there is no transaction in progress
2026-09-26 07:03:30,999 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 07:03:31,000 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 07:03:31,224 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 07:03:31,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 07:03:31,486 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 07:03:31,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 07:03:31,736 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 07:03:31,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 07:03:31,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 07:03:32,227 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t1mc.c.gcp-upstream.internal';"
2026-09-26 07:03:32,447 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 07:03:32,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 07:03:32,682 [I] DB_PATCHES have already been applied
I20260926 07:03:32.687738 1893 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktn8ZA6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 07:03:32.687814 1893 mini_ranger.cc:193] Using host: 127.1.217.84
I20260926 07:03:32.691308 1893 mini_ranger.cc:227] Ranger admin URL: http://127.1.217.84:43091
Sep 26, 2026 7:03:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:03:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:03:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 7:03:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 7:03:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 7:03:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 7:03:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 07:03:33.779583 1893 mini_ranger.cc:160] Time spent starting Ranger: real 8.281s user 0.000s sys 0.009s
Sep 26, 2026 7:03:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 7:03:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 7:03:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 7:03:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 7:03:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 07:03:55.251911 1893 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.1.217.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.1.217.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangerkms_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.1.217.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/rangerkms_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.1.217.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.1.217.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.1.217.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/HTTP_127.1.217.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 07:03:55 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 07:03:55.329437 1893 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 07:03:55.338464 1893 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 07:03:56.196834 1893 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 07:03:56.196913 1893 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms
I20260926 07:03:56.196934 1893 mini_ranger_kms.cc:227] Using postgres at 127.1.217.84:47619
2026-09-26 07:03:56,357 [I] DB FLAVOR :POSTGRES
2026-09-26 07:03:56,357 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 07:03:56,357 [I] Checking connection
2026-09-26 07:03:56,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 07:03:56,608 [I] connection success
2026-09-26 07:03:56,608 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 07:03:56,608 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 07:03:56,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 07:03:56,835 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 07:03:56,835 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 07:03:56,835 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 07:03:56,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.1.217.84:47619/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 07:03:57,094 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 07:03:57.546651 1893 mini_ranger_kms.cc:310] Created kms service
I20260926 07:03:57.737150 1893 mini_ranger_kms.cc:326] Created kudu user
I20260926 07:03:57.780803 1893 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 07:03:57 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406235, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:03:58.148546 1893 mini_ranger_kms.cc:384] Added ranger policy
I20260926 07:03:58.148666 1893 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktn8ZA6/build/release/bin/postgresql.jar:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktn8ZA6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 07:03:58.148685 1893 mini_ranger_kms.cc:240] Using host: 127.1.217.84
I20260926 07:03:58.152109 1893 mini_ranger_kms.cc:276] Ranger KMS PID: 14019
I20260926 07:03:58.152190 1893 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.1.217.84:58577
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 7:03:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 07:04:00.407472 1893 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.078s user 0.009s sys 0.135s
I20260926 07:04:00.407552 1893 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 07:04:00.407588 1893 mini_ranger_kms.cc:401] 127.1.217.84:58577/kms/v1/keys
Sep 26 07:04:00 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790406240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 07:04:00 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
Sep 26 07:04:01 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406240, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.1.217.84@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
Sep 26 07:04:03 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:04:03.595103 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39387
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:39387
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:03.676738 14070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:03.676996 14070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:03.677021 14070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:03.678224 14070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:03.678275 14070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:03.678290 14070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:03.678303 14070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:03.678315 14070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:03.680272 14070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
--ranger_kms_url=127.1.217.84:58577/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:39387
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:03.680564 14070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:03.680838 14070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:04:03.686106 14070 server_base.cc:1034] running on GCE node
W20260926 07:04:03.686239 14078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:03.686254 14075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:03.686522 14076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:03.686744 14070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:03.687101 14070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:03.688263 14070 hybrid_clock.cc:648] HybridClock initialized: now 1790406243688246 us; error 34 us; skew 500 ppm
Sep 26 07:04:03 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:03.691298 14070 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:03.691810 14070 webserver.cc:460] Webserver started at http://127.1.217.126:35871/ using document root <none> and password file <none>
I20260926 07:04:03.692075 14070 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:03.692116 14070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:03.692256 14070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:04:03 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406243, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:03.749552 14070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "519987451ad04fbc99ea5ac607fa363d"
format_stamp: "Formatted at 2026-09-26 07:04:03 on dist-test-slave-t1mc"
server_key: "255f141d6240480b0bee6bcf8e62703e"
server_key_iv: "1ece727bf2b33b32c4bb11086cbf0527"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:03.750015 14070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "519987451ad04fbc99ea5ac607fa363d"
format_stamp: "Formatted at 2026-09-26 07:04:03 on dist-test-slave-t1mc"
server_key: "255f141d6240480b0bee6bcf8e62703e"
server_key_iv: "1ece727bf2b33b32c4bb11086cbf0527"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:03.751837 14070 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:03.770087 14085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:03.770378 14070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:04:03.770495 14070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "519987451ad04fbc99ea5ac607fa363d"
format_stamp: "Formatted at 2026-09-26 07:04:03 on dist-test-slave-t1mc"
server_key: "255f141d6240480b0bee6bcf8e62703e"
server_key_iv: "1ece727bf2b33b32c4bb11086cbf0527"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:03.770619 14070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:03.810823 14070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:03.812907 14070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:03.813067 14070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:03.817597 14070 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:39387
I20260926 07:04:03.817664 14137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:39387 every 8 connection(s)
I20260926 07:04:03.818032 14070 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:03.818760 14138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:03.821105 14138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Bootstrap starting.
I20260926 07:04:03.821455 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14070
I20260926 07:04:03.821583 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:04:03.821772 1893 external_mini_cluster.cc:1428] Setting key 0f753e37486a622121c441e5a4485a14
I20260926 07:04:03.821908 14138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:03.822436 14138 log.cc:824] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:03.823179 14138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: No bootstrap required, opened a new log
I20260926 07:04:03.825035 14138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:03.825191 14138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:03.825251 14138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519987451ad04fbc99ea5ac607fa363d, State: Initialized, Role: FOLLOWER
I20260926 07:04:03.825388 14138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:03.825481 14138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:03.825533 14138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:03.825605 14138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:03.826146 14138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:03.826258 14138 leader_election.cc:302] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519987451ad04fbc99ea5ac607fa363d; no voters:
I20260926 07:04:03.826468 14138 leader_election.cc:288] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:04:03.826496 14143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:03.826696 14143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 LEADER]: Becoming Leader. State: Replica: 519987451ad04fbc99ea5ac607fa363d, State: Running, Role: LEADER
I20260926 07:04:03.826824 14138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:03.826819 14143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:03.827129 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "519987451ad04fbc99ea5ac607fa363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } } }
I20260926 07:04:03.827260 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:03.827431 14143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519987451ad04fbc99ea5ac607fa363d. Latest consensus state: current_term: 1 leader_uuid: "519987451ad04fbc99ea5ac607fa363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } } }
I20260926 07:04:03.827553 14143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:03.827929 14147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:03.828490 14147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:03.829063 14138 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:04:03.830361 14147 catalog_manager.cc:1347] Generated new cluster ID: 183977b7c86b4e208e3b9b7d88b65d1b
I20260926 07:04:03.830410 14147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:04:03 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:03.835464 14141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:03.822307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.1.217.126:39387)
0926 07:04:03.822498 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:03.822508 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:03.822544 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:04:03.823075 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:03.823084 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:03.823163 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:03.824079 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:03.824187 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:03.825573 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:03.826655 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:03.828021 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:03.828143 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:03.831793 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:03.831797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:03.831797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:03.831809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:03.833682 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:03.834051 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:03.834054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:03.834054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:03.834141 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:03.834313 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:03.834314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:03.834314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:03.834591 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:03.834745 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:03.835154 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 07:04:03.835262 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260926 07:04:03.856508 14147 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:04:03.857422 14147 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:03.876613 14147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Generated new TSK 0
I20260926 07:04:03.876914 14147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:04:04.603662 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:04.687954 14183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:04.688148 14183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:04.688170 14183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:04.689390 14183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:04.689431 14183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:04.689458 14183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:04.691144 14183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:04.691372 14183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:04.691617 14183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:04.696825 14191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:04.696825 14189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:04.697109 14188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:04.697809 14183 server_base.cc:1034] running on GCE node
I20260926 07:04:04.698027 14183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:04.698302 14183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:04.699455 14183 hybrid_clock.cc:648] HybridClock initialized: now 1790406244699447 us; error 32 us; skew 500 ppm
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:04.702040 14183 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:04.702450 14183 webserver.cc:460] Webserver started at http://127.1.217.65:37645/ using document root <none> and password file <none>
I20260926 07:04:04.702701 14183 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:04.702770 14183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:04.702903 14183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:04.716737 14183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "7eaa303776d74567a8664d24ff593a5e"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "93bc2d649b3bab1330df24e0fc8d9178"
server_key_iv: "2e01fd320e2b9d69db368c110349d4c2"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.717336 14183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "7eaa303776d74567a8664d24ff593a5e"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "93bc2d649b3bab1330df24e0fc8d9178"
server_key_iv: "2e01fd320e2b9d69db368c110349d4c2"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.727502 14183 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.001s sys 0.000s
I20260926 07:04:04.741379 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:04.741639 14183 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:04.741746 14183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "7eaa303776d74567a8664d24ff593a5e"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "93bc2d649b3bab1330df24e0fc8d9178"
server_key_iv: "2e01fd320e2b9d69db368c110349d4c2"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.741925 14183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:04.752604 14183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:04.754990 14183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:04.755120 14183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:04.755334 14183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:04.755681 14183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:04.755719 14183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:04.755749 14183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:04.755769 14183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:04.762200 14183 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:33967
I20260926 07:04:04.762686 14183 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:04.763648 14313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:33967 every 8 connection(s)
I20260926 07:04:04.768850 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14183
I20260926 07:04:04.768955 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:04:04.769150 1893 external_mini_cluster.cc:1428] Setting key b996074eb11181391af50ecad6a7bb52
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:04.775538 14317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:04.764379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.65:55541)
0926 07:04:04.764643 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:04.764684 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:04.765323 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:04.765715 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:04.765720 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:04.765954 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:04.766216 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:04.766222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:04.767266 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:04.767267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:04.768536 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:04.768539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:04.769287 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:04.769288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:04.769434 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:04.770073 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:04.770085 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:04.771778 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:04.773892 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:04.773898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:04.773900 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:04.774190 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:04.774412 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:04.774412 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:04.774412 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:04.774551 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:04.774876 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:04.774882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:04.775039 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 07:04:04.775144 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 07:04:04.775291 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":86,"threads_started":1}
I20260926 07:04:04.776237 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:04.764353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:55541 (local address 127.1.217.126:39387)
0926 07:04:04.764634 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:04.764636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:04.764805 (+ 169us) server_negotiation.cc:407] Connection header received
0926 07:04:04.765490 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:04.765491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:04.765506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:04.765581 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:04.766327 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:04.767158 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:04.768640 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:04.769076 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:04.769147 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:04.771887 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:04.771889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:04.771889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:04.771896 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:04.773799 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:04.774280 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:04.774281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:04.774281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:04.774342 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:04.774634 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:04.774635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:04.774635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:04.774787 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:04.774850 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:04.776054 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0926 07:04:04.776139 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":81,"threads_started":1}
I20260926 07:04:04.777048 14314 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:04.777139 14314 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:04.777318 14314 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:04.777961 14102 ts_manager.cc:194] Registered new tserver with Master: 7eaa303776d74567a8664d24ff593a5e (127.1.217.65:33967)
I20260926 07:04:04.778848 14102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:55541
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:04:04.812651 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:04.932951 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:04.933228 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:04.933271 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:04.935034 14320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:04.935099 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:04.935140 14320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:04.937407 14320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:04.937783 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:04.938078 14320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:04.944303 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:04.944555 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:04.944543 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:04.945467 14320 server_base.cc:1034] running on GCE node
I20260926 07:04:04.945716 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:04.945955 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:04.947108 14320 hybrid_clock.cc:648] HybridClock initialized: now 1790406244947087 us; error 33 us; skew 500 ppm
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:04.950367 14320 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:04.950842 14320 webserver.cc:460] Webserver started at http://127.1.217.66:37147/ using document root <none> and password file <none>
I20260926 07:04:04.951094 14320 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:04.951145 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:04.951265 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:04:04 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:04.963864 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "28842c2594fe482eb22de6ad51f38033"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "99cb7c4d7866b4b65042659eb58e5a56"
server_key_iv: "b11d605b6f80f06621ee4997918b4023"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.964299 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "28842c2594fe482eb22de6ad51f38033"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "99cb7c4d7866b4b65042659eb58e5a56"
server_key_iv: "b11d605b6f80f06621ee4997918b4023"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.972990 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.003s
I20260926 07:04:04.990223 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:04.990736 14320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:04.990841 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "28842c2594fe482eb22de6ad51f38033"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "99cb7c4d7866b4b65042659eb58e5a56"
server_key_iv: "b11d605b6f80f06621ee4997918b4023"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:04.990943 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:05.007943 14320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:05.010481 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:05.010707 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:05.010977 14320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:05.011397 14320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:05.011489 14320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.011564 14320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:05.011626 14320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.017987 14320 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:42229
I20260926 07:04:05.018431 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:05.020205 14451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:42229 every 8 connection(s)
I20260926 07:04:05.023113 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14320
I20260926 07:04:05.023224 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:04:05.023407 1893 external_mini_cluster.cc:1428] Setting key b3e15667524c9e9c7a684fb49fa4707c
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406244, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:05.032222 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.021411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:60819 (local address 127.1.217.126:39387)
0926 07:04:05.021458 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:05.021460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:05.021686 (+ 226us) server_negotiation.cc:407] Connection header received
0926 07:04:05.022401 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:05.022403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:05.022417 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:05.022449 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:05.023328 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.024107 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.025644 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.025972 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.026020 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.028792 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:05.028793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:05.028793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:05.028802 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:05.030649 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.031139 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.031140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.031140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.031194 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.031452 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.031453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.031453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.031583 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:05.031623 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:05.032011 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 07:04:05.032108 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:04:05.033120 14454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.021134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.66:60819)
0926 07:04:05.021554 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:05.021596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:05.022268 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:05.022638 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:05.022643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:05.022884 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:05.023170 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.023177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.024229 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.024230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.025536 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.025539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.026113 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.026114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.026262 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.026866 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:05.026876 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:05.028635 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:05.030755 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.030762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.030765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.031056 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.031262 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.031263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.031263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.031384 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.031677 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:05.031682 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:05.031843 (+ 161us) client_negotiation.cc:769] Sending connection context
0926 07:04:05.032644 (+ 801us) client_negotiation.cc:240] Negotiation successful
0926 07:04:05.032867 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":84,"threads_started":1}
I20260926 07:04:05.033820 14452 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:05.033915 14452 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.034111 14452 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:05.034552 14102 ts_manager.cc:194] Registered new tserver with Master: 28842c2594fe482eb22de6ad51f38033 (127.1.217.66:42229)
I20260926 07:04:05.035073 14102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:60819
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:04:05.068608 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:05.194254 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:05.194633 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:05.194787 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:05.196699 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:05.196853 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:05.196947 14457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:05.199224 14457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:05.199719 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:05.200093 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:05.207278 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.207661 14457 server_base.cc:1034] running on GCE node
W20260926 07:04:05.208107 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.208665 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.209002 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:05.209280 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:05.210464 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790406245210441 us; error 36 us; skew 500 ppm
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:05.213950 14457 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:05.214475 14457 webserver.cc:460] Webserver started at http://127.1.217.67:43855/ using document root <none> and password file <none>
I20260926 07:04:05.214744 14457 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:05.214802 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:05.214936 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:05.233672 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "cc235223d674493aa0d8e76a92cad2d0"
format_stamp: "Formatted at 2026-09-26 07:04:05 on dist-test-slave-t1mc"
server_key: "7513a7f31f957e41e346e8a5417d0f0a"
server_key_iv: "7c4682a1ae877790d496ff822ff2f45f"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.234120 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "cc235223d674493aa0d8e76a92cad2d0"
format_stamp: "Formatted at 2026-09-26 07:04:05 on dist-test-slave-t1mc"
server_key: "7513a7f31f957e41e346e8a5417d0f0a"
server_key_iv: "7c4682a1ae877790d496ff822ff2f45f"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.235893 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 07:04:05.263377 14472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.263609 14457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:05.263695 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "cc235223d674493aa0d8e76a92cad2d0"
format_stamp: "Formatted at 2026-09-26 07:04:05 on dist-test-slave-t1mc"
server_key: "7513a7f31f957e41e346e8a5417d0f0a"
server_key_iv: "7c4682a1ae877790d496ff822ff2f45f"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.263782 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:05.290689 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:05.293283 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:05.293484 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:05.293738 14457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:05.294142 14457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:05.294220 14457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.294304 14457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:05.294359 14457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.300647 14457 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:43297
I20260926 07:04:05.301152 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:04:05.311110 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14457
I20260926 07:04:05.311231 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:04:05.311437 1893 external_mini_cluster.cc:1428] Setting key 5f398dd935bf546bc96cc28f6b572520
I20260926 07:04:05.314751 14585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:43297 every 8 connection(s)
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:05.326884 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.315919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:36103 (local address 127.1.217.126:39387)
0926 07:04:05.315960 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:05.315962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:05.316388 (+ 426us) server_negotiation.cc:407] Connection header received
0926 07:04:05.317152 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:05.317153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:05.317164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:05.317195 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:05.318083 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.318855 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.320337 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.320642 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.320687 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.323436 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:05.323438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:05.323438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:05.323446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:05.325234 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.325726 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.325727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.325728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.325797 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.326101 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.326102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.326103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.326240 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:05.326285 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:05.326674 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 07:04:05.326757 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:04:05.327672 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.315615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.67:36103)
0926 07:04:05.316253 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:05.316304 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:05.317029 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:05.317401 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:05.317405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:05.317667 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:05.317970 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.317976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.318973 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.318974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.320230 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.320233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.320824 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.320825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.320965 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.321604 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:05.321615 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:05.323321 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:05.325334 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.325341 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.325343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.325633 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.325877 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.325878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.325878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.326017 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.326338 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:05.326344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:05.326511 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 07:04:05.327262 (+ 751us) client_negotiation.cc:240] Negotiation successful
0926 07:04:05.327429 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":108,"threads_started":1}
I20260926 07:04:05.328366 14586 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:05.328460 14586 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.328647 14586 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:05.329087 14102 ts_manager.cc:194] Registered new tserver with Master: cc235223d674493aa0d8e76a92cad2d0 (127.1.217.67:43297)
I20260926 07:04:05.329571 14102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:36103
I20260926 07:04:05.333940 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:05.334662 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14183
I20260926 07:04:05.342578 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14320
I20260926 07:04:05.349115 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14457
I20260926 07:04:05.355940 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14070
I20260926 07:04:05.360834 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39387
--webserver_interface=127.1.217.126
--webserver_port=35871
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:39387
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:05.442351 14589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:05.442607 14589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:05.442646 14589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:05.443804 14589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:05.443872 14589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:05.443898 14589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:05.443929 14589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:05.443962 14589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:05.445590 14589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:39387
--ranger_config_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:39387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=35871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:05.445858 14589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:05.446079 14589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:05.450789 14597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.450853 14594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.450794 14595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.451126 14589 server_base.cc:1034] running on GCE node
I20260926 07:04:05.451381 14589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:05.451695 14589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:05.452839 14589 hybrid_clock.cc:648] HybridClock initialized: now 1790406245452826 us; error 28 us; skew 500 ppm
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:05.455736 14589 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:05.456185 14589 webserver.cc:460] Webserver started at http://127.1.217.126:35871/ using document root <none> and password file <none>
I20260926 07:04:05.456414 14589 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:05.456489 14589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:05.516840 14589 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:05.517560 14605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.517720 14589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:05.517823 14589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "519987451ad04fbc99ea5ac607fa363d"
format_stamp: "Formatted at 2026-09-26 07:04:03 on dist-test-slave-t1mc"
server_key: "255f141d6240480b0bee6bcf8e62703e"
server_key_iv: "1ece727bf2b33b32c4bb11086cbf0527"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.518174 14589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:05.527462 14589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:05.529047 14589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:05.529160 14589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:05.533277 14589 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:39387
I20260926 07:04:05.533313 14657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:39387 every 8 connection(s)
I20260926 07:04:05.533676 14589 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:05.534816 14658 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:04:05.535470 14658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Bootstrap starting.
I20260926 07:04:05.536345 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14589
I20260926 07:04:05.536625 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:33967
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:39387
--webserver_port=37645
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:05.543327 14658 log.cc:824] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:05.545037 14658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:05.545312 14658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Bootstrap complete.
I20260926 07:04:05.547608 14658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:05.547852 14658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519987451ad04fbc99ea5ac607fa363d, State: Initialized, Role: FOLLOWER
I20260926 07:04:05.548055 14658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:05.548132 14658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:05.548184 14658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:05.548235 14658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:04:05.549293 14658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:05.549427 14658 leader_election.cc:302] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 519987451ad04fbc99ea5ac607fa363d; no voters:
I20260926 07:04:05.549623 14658 leader_election.cc:288] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:04:05.549655 14662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:04:05.549762 14662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [term 2 LEADER]: Becoming Leader. State: Replica: 519987451ad04fbc99ea5ac607fa363d, State: Running, Role: LEADER
I20260926 07:04:05.549894 14658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:05.549923 14662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } }
I20260926 07:04:05.550230 14662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 519987451ad04fbc99ea5ac607fa363d. Latest consensus state: current_term: 2 leader_uuid: "519987451ad04fbc99ea5ac607fa363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } } }
I20260926 07:04:05.550235 14663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "519987451ad04fbc99ea5ac607fa363d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "519987451ad04fbc99ea5ac607fa363d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 39387 } } }
I20260926 07:04:05.550340 14662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:05.550802 14663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:05.551393 14658 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 07:04:05.553509 14670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:05.554469 14670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:05.554725 14670 catalog_manager.cc:1259] Loaded cluster ID: 183977b7c86b4e208e3b9b7d88b65d1b
I20260926 07:04:05.554805 14670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:04:05.556849 14670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:05.557050 14670 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Loaded TSK: 0
I20260926 07:04:05.557278 14670 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:04:05.646024 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:05.646296 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:05.646334 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:05.647981 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:05.648074 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:05.648150 14660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:05.650348 14660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:33967
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=37645
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:05.650569 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:05.650844 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:05.657469 14695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.657475 14698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.657496 14660 server_base.cc:1034] running on GCE node
W20260926 07:04:05.657745 14696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.658018 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:05.658294 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:05.661613 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790406245661591 us; error 43 us; skew 500 ppm
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:05.664937 14660 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:05.665470 14660 webserver.cc:460] Webserver started at http://127.1.217.65:37645/ using document root <none> and password file <none>
I20260926 07:04:05.665738 14660 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:05.665836 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:05.700506 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 07:04:05.701545 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.701709 14660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:05.701792 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "7eaa303776d74567a8664d24ff593a5e"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "93bc2d649b3bab1330df24e0fc8d9178"
server_key_iv: "2e01fd320e2b9d69db368c110349d4c2"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.702267 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:05.750993 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:05.753377 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:05.753544 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:05.753820 14660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:05.754174 14660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:05.754242 14660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.754282 14660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:05.754307 14660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.761647 14660 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:33967
I20260926 07:04:05.762140 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:05.763295 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:33967 every 8 connection(s)
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:05.772061 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14660
I20260926 07:04:05.772358 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:42229
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:39387
--webserver_port=37147
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:05.775583 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.764372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:52561 (local address 127.1.217.126:39387)
0926 07:04:05.765646 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:05.765655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:05.765697 (+ 42us) server_negotiation.cc:407] Connection header received
0926 07:04:05.765846 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:05.765852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:05.765919 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:05.766061 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:05.766845 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.767827 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.768882 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.769368 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.769486 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.771673 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:05.771677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:05.771678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:05.771689 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:05.773564 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.774141 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.774145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.774145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.774262 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.774564 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.774565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.774565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.774851 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:05.774954 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:05.775250 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 07:04:05.775367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1212,"thread_start_us":78,"threads_started":1}
I20260926 07:04:05.775597 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.763994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.65:52561)
0926 07:04:05.764701 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:05.764748 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:05.765415 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:05.766145 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:05.766150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:05.766421 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:05.766712 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.766719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.767951 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.767952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.768755 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.768757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.769485 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.769487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.769611 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.770210 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:05.770218 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:05.771554 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:05.773673 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.773681 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.773683 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.774014 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.774334 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.774335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.774335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.774466 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.774944 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:05.774950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:05.775087 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 07:04:05.775200 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:04:05.775374 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":82,"threads_started":1}
I20260926 07:04:05.776793 14820 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:05.776886 14820 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.777070 14820 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:05.778005 14820 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:39387 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:04:05.778076 14820 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.778141 14820 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:05.778566 14820 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.778645 14820 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:05.778941 14820 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:04:05.882159 14824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:05.882413 14824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:05.882472 14824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:05.883858 14824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:05.883915 14824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:05.883955 14824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:05.885695 14824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:42229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=37147
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:05.885946 14824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:05.886215 14824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:05.891290 14832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.891404 14829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:05.891536 14830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:05.891592 14824 server_base.cc:1034] running on GCE node
I20260926 07:04:05.891884 14824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:05.892171 14824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:05.893357 14824 hybrid_clock.cc:648] HybridClock initialized: now 1790406245893374 us; error 57 us; skew 500 ppm
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:05.895982 14824 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:05.896526 14824 webserver.cc:460] Webserver started at http://127.1.217.66:37147/ using document root <none> and password file <none>
I20260926 07:04:05.896816 14824 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:05.896931 14824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:05.917629 14824 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260926 07:04:05.918581 14840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.918792 14824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:05.918895 14824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "28842c2594fe482eb22de6ad51f38033"
format_stamp: "Formatted at 2026-09-26 07:04:04 on dist-test-slave-t1mc"
server_key: "99cb7c4d7866b4b65042659eb58e5a56"
server_key_iv: "b11d605b6f80f06621ee4997918b4023"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:05.919272 14824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:05.944326 14824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:05.946775 14824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:05.946918 14824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:05.947141 14824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:05.947461 14824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:05.947502 14824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.947546 14824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:05.947567 14824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:05.954807 14824 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:42229
I20260926 07:04:05.955300 14824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:05.955367 14953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:42229 every 8 connection(s)
Sep 26 07:04:05 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406245, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:05.964475 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14824
I20260926 07:04:05.964798 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.1.217.84:58577/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:43297
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:39387
--webserver_port=43855
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:05.966291 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.956371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:35471 (local address 127.1.217.126:39387)
0926 07:04:05.956419 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:05.956421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:05.956679 (+ 258us) server_negotiation.cc:407] Connection header received
0926 07:04:05.957226 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:05.957227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:05.957242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:05.957298 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:05.957822 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.958347 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.959339 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.959558 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.959643 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.962727 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:05.962728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:05.962729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:05.962737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:05.964387 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.965209 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.965210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.965210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.965268 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:05.965563 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:05.965564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:05.965564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:05.965698 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:05.965753 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:05.966110 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 07:04:05.966197 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:04:05.967022 14956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:05.956388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.66:35471)
0926 07:04:05.956548 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:05.956587 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:05.957122 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:05.957370 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:05.957373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:05.957541 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:05.957725 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.957729 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.958443 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.958444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.959238 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:05.959240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:05.959672 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:05.959674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:05.959810 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:05.960391 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:05.960402 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:05.962614 (+ 2212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:05.964481 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.964487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.964489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.965116 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.965341 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:05.965341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:05.965342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:05.965467 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:05.965787 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:05.965793 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:05.965946 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 07:04:05.966627 (+ 681us) client_negotiation.cc:240] Negotiation successful
0926 07:04:05.966799 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260926 07:04:05.967645 14954 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:05.967731 14954 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.967913 14954 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:05.968530 14954 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:39387 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:04:05.968595 14954 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.968652 14954 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:05.969084 14954 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:05.969152 14954 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:05.969475 14954 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:04:06.078994 14957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:06.079197 14957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:06.079221 14957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:06.080500 14957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:06.080549 14957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:06.080577 14957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:06.082219 14957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.1.217.84:58577/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:43297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=43855
--tserver_master_addrs=127.1.217.126:39387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:06.082466 14957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:06.082682 14957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:06.088850 14964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:06.089035 14966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:06.089181 14963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:06.090011 14957 server_base.cc:1034] running on GCE node
I20260926 07:04:06.090217 14957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:06.090487 14957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:06.091665 14957 hybrid_clock.cc:648] HybridClock initialized: now 1790406246091648 us; error 29 us; skew 500 ppm
Sep 26 07:04:06 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:06.094314 14957 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:06.094817 14957 webserver.cc:460] Webserver started at http://127.1.217.67:43855/ using document root <none> and password file <none>
I20260926 07:04:06.095059 14957 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:06.095139 14957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 07:04:06 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:06.114939 14957 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:06.115731 14974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:06.115873 14957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:06.115949 14957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "cc235223d674493aa0d8e76a92cad2d0"
format_stamp: "Formatted at 2026-09-26 07:04:05 on dist-test-slave-t1mc"
server_key: "7513a7f31f957e41e346e8a5417d0f0a"
server_key_iv: "7c4682a1ae877790d496ff822ff2f45f"
server_key_version: "kuduclusterkey@0"
I20260926 07:04:06.117202 14957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:06.143512 14957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:06.145579 14957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:06.145750 14957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:06.146006 14957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:06.146359 14957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:06.146411 14957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:06.146453 14957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:06.146479 14957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:06.153579 14957 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:43297
I20260926 07:04:06.153669 15087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:43297 every 8 connection(s)
I20260926 07:04:06.154042 14957 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:04:06.155380 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 14957
Sep 26 07:04:06 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:06.165256 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:06.154821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:53613 (local address 127.1.217.126:39387)
0926 07:04:06.154878 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:06.154880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:06.155141 (+ 261us) server_negotiation.cc:407] Connection header received
0926 07:04:06.155790 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:06.155791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:06.155805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:06.155837 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:06.156637 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.157425 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.158884 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.159181 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.159249 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:06.161746 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:06.161747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:06.161748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:06.161754 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:06.163748 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:06.164190 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:06.164191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:06.164191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:06.164237 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:06.164510 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:06.164511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:06.164511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:06.164616 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:06.164651 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:06.165058 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 07:04:06.165138 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":28}
I20260926 07:04:06.165364 15091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:06.155780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.1.217.126:39387)
0926 07:04:06.155876 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:06.155878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:06.155996 (+ 118us) server_negotiation.cc:407] Connection header received
0926 07:04:06.156079 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:06.156079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:06.156102 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:06.156177 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:06.156808 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.157469 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.158243 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.158461 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.158545 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:06.160336 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:06.160338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:06.160338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:06.160345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:06.163711 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:06.164105 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:06.164106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:06.164107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:06.164167 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:06.164375 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:06.164376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:06.164376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:06.164507 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:06.164651 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:06.165135 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 07:04:06.165198 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":50,"threads_started":1}
I20260926 07:04:06.166134 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:06.154791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:39387 (local address 127.1.217.67:53613)
0926 07:04:06.154977 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:06.155028 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:06.155658 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:06.156027 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:06.156031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:06.156263 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:06.156537 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:06.156544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.157523 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.157524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:06.158795 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:06.158798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:06.159326 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:06.159328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:06.159477 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:06.160014 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:06.160025 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:06.161645 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:06.163821 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:06.163827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:06.163829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:06.164110 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:06.164310 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:06.164311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:06.164311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:06.164434 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:06.164701 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:06.164706 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:06.164894 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 07:04:06.165808 (+ 914us) client_negotiation.cc:240] Negotiation successful
0926 07:04:06.165971 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260926 07:04:06.166567 15088 heartbeater.cc:347] Connected to a master server at 127.1.217.126:39387
I20260926 07:04:06.166692 15088 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:06.166882 15088 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:06.167492 15088 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:39387 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 07:04:06.167574 15088 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:06.167651 15088 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:06.167922 15088 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:06.167994 15088 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
W20260926 07:04:06.168290 15088 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 07:04:06 dist-test-slave-t1mc krb5kdc[13626](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:06.779242 14820 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:06.779436 14820 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:06.779842 14622 ts_manager.cc:194] Registered new tserver with Master: 7eaa303776d74567a8664d24ff593a5e (127.1.217.65:33967)
I20260926 07:04:06.780519 14622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:52561
I20260926 07:04:06.969645 14954 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:06.969796 14954 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:06.970085 14622 ts_manager.cc:194] Registered new tserver with Master: 28842c2594fe482eb22de6ad51f38033 (127.1.217.66:42229)
I20260926 07:04:06.970496 14622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:35471
Sep 26 07:04:06 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:07.168493 15088 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:07.168648 15088 heartbeater.cc:511] Master 127.1.217.126:39387 requested a full tablet report, sending...
I20260926 07:04:07.169046 14622 ts_manager.cc:194] Registered new tserver with Master: cc235223d674493aa0d8e76a92cad2d0 (127.1.217.67:43297)
I20260926 07:04:07.169541 14622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:53613
I20260926 07:04:07.170964 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:07.182718 15109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.171856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.1.217.126:39387)
0926 07:04:07.172139 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.172141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.172342 (+ 201us) server_negotiation.cc:407] Connection header received
0926 07:04:07.172364 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.172365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.172376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.172426 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:07.173106 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.173841 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.174814 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.175088 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.175139 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.179712 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:07.179714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:07.179714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:07.179721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:07.181379 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:07.181733 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:07.181734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:07.181735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:07.181786 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:07.181949 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:07.181949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:07.181950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:07.182063 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:07.182143 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.182549 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.182623 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":60,"threads_started":1}
I20260926 07:04:07.184893 14622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:04:07.186996 14622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:04:07.200189 15118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.190881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:33967 (local address 127.0.0.1:37394)
0926 07:04:07.191636 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.191644 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:07.191672 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:07.192049 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:07.192052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:07.192054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:07.192117 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.192118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.195155 (+ 3037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.195156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.196610 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.196613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.199907 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.199908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.199991 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.200007 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:04:07.200057 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:04:07.200106 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":62,"threads_started":1}
I20260926 07:04:07.200911 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.190973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.1.217.65:33967)
0926 07:04:07.191887 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.191892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.191901 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:07.191934 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.191936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.191950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.192180 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:07.192288 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.193815 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.196709 (+ 2894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.197420 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.200648 (+ 3228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.200726 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.200753 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.200836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":880,"thread_start_us":46,"threads_started":1}
I20260926 07:04:07.201865 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.190621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:42229 (local address 127.0.0.1:36648)
0926 07:04:07.191458 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.191497 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:07.191551 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:07.193895 (+ 2344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:07.193896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:07.193898 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:07.193942 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.193943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.197538 (+ 3595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.197540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.198979 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.198983 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.201653 (+ 2670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.201653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.201719 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.201724 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:04:07.201754 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 07:04:07.201793 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":83,"threads_started":1}
I20260926 07:04:07.201782 14736 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 (DEFAULT_TABLE table=test-table [id=8607192cf626427187cc1d27cde8ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:07.202176 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1057de0a643878ac35cc6a4c5b4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:07.202497 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.190712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.1.217.66:42229)
0926 07:04:07.193307 (+ 2595us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.193311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.193321 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:04:07.193353 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.193355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.193368 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.194008 (+ 640us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:07.194078 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.195046 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.199075 (+ 4029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.199791 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.202206 (+ 2415us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.202342 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.202368 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.202411 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2562,"thread_start_us":61,"threads_started":1}
I20260926 07:04:07.203279 14888 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 (DEFAULT_TABLE table=test-table [id=8607192cf626427187cc1d27cde8ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:07.203605 14888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1057de0a643878ac35cc6a4c5b4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:07.213035 15123 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Bootstrap starting.
I20260926 07:04:07.213721 15123 tablet_bootstrap.cc:654] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:07.214037 15123 log.cc:824] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:07.214387 15122 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Bootstrap starting.
I20260926 07:04:07.214998 15122 tablet_bootstrap.cc:654] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:07.215269 15122 log.cc:824] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:07.215497 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.191063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:43297 (local address 127.0.0.1:34090)
0926 07:04:07.191756 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.191763 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:07.191790 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:07.198927 (+ 7137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:07.198927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:07.198931 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:07.198999 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.199001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.212876 (+ 13875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.212877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.214091 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.214096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.215203 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.215204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.215276 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.215282 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:04:07.215353 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:04:07.215400 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":51,"threads_started":1}
I20260926 07:04:07.215817 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.191175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.1.217.67:43297)
0926 07:04:07.198620 (+ 7445us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.198625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.198639 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:04:07.198698 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.198700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.198717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.198817 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:07.199080 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.200125 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.214216 (+ 14091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.215026 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.215101 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.215165 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.215684 (+ 519us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.215730 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7380,"thread_start_us":64,"threads_started":1}
I20260926 07:04:07.216058 15123 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: No bootstrap required, opened a new log
I20260926 07:04:07.216130 15123 ts_tablet_manager.cc:1397] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:04:07.216447 15022 tablet_service.cc:1467] Processing CreateTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 (DEFAULT_TABLE table=test-table [id=8607192cf626427187cc1d27cde8ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:07.216840 15022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8e1057de0a643878ac35cc6a4c5b4b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:07.218250 15123 raft_consensus.cc:348] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.218415 15123 raft_consensus.cc:374] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:07.218453 15123 raft_consensus.cc:729] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28842c2594fe482eb22de6ad51f38033, State: Initialized, Role: FOLLOWER
I20260926 07:04:07.218564 15123 consensus_queue.cc:260] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.218801 15123 ts_tablet_manager.cc:1428] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:04:07.219041 14954 heartbeater.cc:503] Master 127.1.217.126:39387 was elected leader, sending a full tablet report...
I20260926 07:04:07.220309 15122 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: No bootstrap required, opened a new log
I20260926 07:04:07.220387 15122 ts_tablet_manager.cc:1397] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 07:04:07.220904 15127 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Bootstrap starting.
I20260926 07:04:07.221517 15127 tablet_bootstrap.cc:654] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:07.221989 15127 log.cc:824] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:07.222667 15127 tablet_bootstrap.cc:492] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: No bootstrap required, opened a new log
I20260926 07:04:07.222730 15127 ts_tablet_manager.cc:1397] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:07.223373 15122 raft_consensus.cc:348] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.223536 15122 raft_consensus.cc:374] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:07.223578 15122 raft_consensus.cc:729] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eaa303776d74567a8664d24ff593a5e, State: Initialized, Role: FOLLOWER
I20260926 07:04:07.223982 15122 consensus_queue.cc:260] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.224247 15122 ts_tablet_manager.cc:1428] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 07:04:07.224361 14820 heartbeater.cc:503] Master 127.1.217.126:39387 was elected leader, sending a full tablet report...
I20260926 07:04:07.225137 15127 raft_consensus.cc:348] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.225293 15127 raft_consensus.cc:374] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:07.225340 15127 raft_consensus.cc:729] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc235223d674493aa0d8e76a92cad2d0, State: Initialized, Role: FOLLOWER
I20260926 07:04:07.225466 15127 consensus_queue.cc:260] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.225706 15127 ts_tablet_manager.cc:1428] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 07:04:07.225831 15088 heartbeater.cc:503] Master 127.1.217.126:39387 was elected leader, sending a full tablet report...
W20260926 07:04:07.265553 14821 tablet.cc:2369] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 07:04:07 dist-test-slave-t1mc krb5kdc[13626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406246, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for HTTP/127.1.217.84@KRBTEST.COM
I20260926 07:04:07.321395 15126 raft_consensus.cc:482] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:07.321565 15126 raft_consensus.cc:504] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.322014 15126 leader_election.cc:288] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc235223d674493aa0d8e76a92cad2d0 (127.1.217.67:43297), 7eaa303776d74567a8664d24ff593a5e (127.1.217.65:33967)
I20260926 07:04:07.327780 15121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.322351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50323 (local address 127.1.217.67:43297)
0926 07:04:07.322406 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.322408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.322871 (+ 463us) server_negotiation.cc:407] Connection header received
0926 07:04:07.323024 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.323025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.323035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.323061 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:07.323595 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.324312 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.326074 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.326581 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.326625 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.326634 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.327661 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.327710 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:04:07.328483 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.322297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:43297 (local address 127.1.217.66:50323)
0926 07:04:07.322706 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.322716 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:07.322935 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:07.323124 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:07.323125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:07.323126 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:07.323203 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.323205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.324545 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.324546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.325982 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.325985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.327517 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.327518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.327582 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.327587 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:04:07.328343 (+ 756us) client_negotiation.cc:240] Negotiation successful
0926 07:04:07.328388 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":108,"threads_started":1}
I20260926 07:04:07.328945 15042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1057de0a643878ac35cc6a4c5b4b7" candidate_uuid: "28842c2594fe482eb22de6ad51f38033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc235223d674493aa0d8e76a92cad2d0" is_pre_election: true
I20260926 07:04:07.329145 15042 raft_consensus.cc:2454] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28842c2594fe482eb22de6ad51f38033 in term 0.
I20260926 07:04:07.329396 14842 leader_election.cc:302] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28842c2594fe482eb22de6ad51f38033, cc235223d674493aa0d8e76a92cad2d0; no voters:
I20260926 07:04:07.329538 15126 raft_consensus.cc:2781] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:07.329589 15126 raft_consensus.cc:482] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:07.329617 15126 raft_consensus.cc:3037] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:07.330523 15126 raft_consensus.cc:504] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.330667 15126 leader_election.cc:288] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [CANDIDATE]: Term 1 election: Requested vote from peers cc235223d674493aa0d8e76a92cad2d0 (127.1.217.67:43297), 7eaa303776d74567a8664d24ff593a5e (127.1.217.65:33967)
I20260926 07:04:07.330896 15042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1057de0a643878ac35cc6a4c5b4b7" candidate_uuid: "28842c2594fe482eb22de6ad51f38033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc235223d674493aa0d8e76a92cad2d0"
I20260926 07:04:07.330973 15042 raft_consensus.cc:3037] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:07.331765 15042 raft_consensus.cc:2454] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28842c2594fe482eb22de6ad51f38033 in term 1.
I20260926 07:04:07.332496 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.322608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:53173 (local address 127.1.217.65:33967)
0926 07:04:07.322647 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.322649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.323499 (+ 850us) server_negotiation.cc:407] Connection header received
0926 07:04:07.324241 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.324243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.324255 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.324283 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:07.324821 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.325505 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.328144 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.328704 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.328752 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.328760 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.332371 (+ 3611us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.332428 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:07.332901 14842 leader_election.cc:302] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28842c2594fe482eb22de6ad51f38033, cc235223d674493aa0d8e76a92cad2d0; no voters:
I20260926 07:04:07.335788 15126 raft_consensus.cc:2781] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:07.335953 15126 raft_consensus.cc:686] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 1 LEADER]: Becoming Leader. State: Replica: 28842c2594fe482eb22de6ad51f38033, State: Running, Role: LEADER
I20260926 07:04:07.336067 15126 consensus_queue.cc:237] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } }
I20260926 07:04:07.336265 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.322538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:33967 (local address 127.1.217.66:53173)
0926 07:04:07.323417 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.323428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:07.323556 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:07.324349 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:07.324350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:07.324352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:07.324425 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.324426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.325661 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.325661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.328050 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:07.328053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.332207 (+ 4154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.332208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:07.332284 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.332288 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:04:07.332650 (+ 362us) client_negotiation.cc:240] Negotiation successful
0926 07:04:07.336181 (+ 3531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":74,"threads_started":1}
I20260926 07:04:07.337018 14622 catalog_manager.cc:5495] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28842c2594fe482eb22de6ad51f38033 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "28842c2594fe482eb22de6ad51f38033" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28842c2594fe482eb22de6ad51f38033" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 } health_report { overall_health: UNKNOWN } } }
I20260926 07:04:07.337879 14774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1057de0a643878ac35cc6a4c5b4b7" candidate_uuid: "28842c2594fe482eb22de6ad51f38033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eaa303776d74567a8664d24ff593a5e" is_pre_election: true
I20260926 07:04:07.338048 14774 raft_consensus.cc:2454] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28842c2594fe482eb22de6ad51f38033 in term 0.
I20260926 07:04:07.338196 14773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8e1057de0a643878ac35cc6a4c5b4b7" candidate_uuid: "28842c2594fe482eb22de6ad51f38033" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eaa303776d74567a8664d24ff593a5e"
I20260926 07:04:07.338259 14773 raft_consensus.cc:3037] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:07.339102 14773 raft_consensus.cc:2454] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28842c2594fe482eb22de6ad51f38033 in term 1.
I20260926 07:04:07.378748 15117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.375325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.1.217.66:42229)
0926 07:04:07.375382 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:07.375383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:07.375391 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:07.375430 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:07.375433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:07.375458 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:07.375558 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:07.375702 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.376440 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.377611 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:07.377914 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:07.377979 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:07.378201 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:07.378341 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:07.378426 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:07.378644 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 07:04:07.378667 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:04:07.380887 14773 raft_consensus.cc:1261] T c8e1057de0a643878ac35cc6a4c5b4b7 P 7eaa303776d74567a8664d24ff593a5e [term 1 FOLLOWER]: Refusing update from remote peer 28842c2594fe482eb22de6ad51f38033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:07.381265 15135 consensus_queue.cc:1035] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eaa303776d74567a8664d24ff593a5e" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 33967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:07.383545 15042 raft_consensus.cc:1261] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 1 FOLLOWER]: Refusing update from remote peer 28842c2594fe482eb22de6ad51f38033: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:07.383837 15135 consensus_queue.cc:1035] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc235223d674493aa0d8e76a92cad2d0" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:07.387748 15138 mvcc.cc:205] Tried to move back new op lower bound from 7333503989268164608 to 7333503989088514048. Current Snapshot: MvccSnapshot[applied={T|T < 7333503989268164608}]
I20260926 07:04:07.388815 15141 mvcc.cc:205] Tried to move back new op lower bound from 7333503989268164608 to 7333503989088514048. Current Snapshot: MvccSnapshot[applied={T|T < 7333503989268164608}]
I20260926 07:04:07.390770 15139 mvcc.cc:205] Tried to move back new op lower bound from 7333503989268164608 to 7333503989088514048. Current Snapshot: MvccSnapshot[applied={T|T < 7333503989268164608}]
I20260926 07:04:07.393381 14622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:07.393491 14622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:07.394207 14622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 519987451ad04fbc99ea5ac607fa363d: Sending DeleteTablet for 3 replicas of tablet c8e1057de0a643878ac35cc6a4c5b4b7
I20260926 07:04:07.394671 14888 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:07 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36648
I20260926 07:04:07.394904 14736 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:07 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:37394
I20260926 07:04:07.395292 15022 tablet_service.cc:1514] Processing DeleteTablet for tablet c8e1057de0a643878ac35cc6a4c5b4b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:07 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:34090
I20260926 07:04:07.395838 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14660
I20260926 07:04:07.399056 15147 tablet_replica.cc:330] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: stopping tablet replica
I20260926 07:04:07.399169 15147 raft_consensus.cc:2229] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:04:07.399305 15147 raft_consensus.cc:2258] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:07.399483 15149 tablet_replica.cc:330] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: stopping tablet replica
I20260926 07:04:07.399588 15149 raft_consensus.cc:2229] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:07.399710 15149 raft_consensus.cc:2258] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:07.400213 15149 ts_tablet_manager.cc:1905] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:07.402272 15147 ts_tablet_manager.cc:1905] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:07.403851 15147 ts_tablet_manager.cc:1918] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:07.403930 15147 log.cc:1196] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/c8e1057de0a643878ac35cc6a4c5b4b7
I20260926 07:04:07.404222 15147 ts_tablet_manager.cc:1939] T c8e1057de0a643878ac35cc6a4c5b4b7 P 28842c2594fe482eb22de6ad51f38033: Deleting consensus metadata
I20260926 07:04:07.408885 15149 ts_tablet_manager.cc:1918] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:07.408972 15149 log.cc:1196] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/c8e1057de0a643878ac35cc6a4c5b4b7
I20260926 07:04:07.409245 15149 ts_tablet_manager.cc:1939] T c8e1057de0a643878ac35cc6a4c5b4b7 P cc235223d674493aa0d8e76a92cad2d0: Deleting consensus metadata
I20260926 07:04:07.409722 14607 catalog_manager.cc:4841] TS cc235223d674493aa0d8e76a92cad2d0 (127.1.217.67:43297): tablet c8e1057de0a643878ac35cc6a4c5b4b7 (table test-table [id=8607192cf626427187cc1d27cde8ec86]) successfully deleted
I20260926 07:04:07.412276 14609 catalog_manager.cc:4841] TS 28842c2594fe482eb22de6ad51f38033 (127.1.217.66:42229): tablet c8e1057de0a643878ac35cc6a4c5b4b7 (table test-table [id=8607192cf626427187cc1d27cde8ec86]) successfully deleted
W20260926 07:04:07.412592 14608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:33967 (error 108)
I20260926 07:04:07.415081 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14824
I20260926 07:04:07.420907 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:07.420728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:33967 (local address 127.0.0.1:58472)
0926 07:04:07.420796 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:07.420865 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:33967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 07:04:07.421355 14608 catalog_manager.cc:4584] TS 7eaa303776d74567a8664d24ff593a5e (127.1.217.65:33967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8e1057de0a643878ac35cc6a4c5b4b7: Network error: Client connection negotiation failed: client connection to 127.1.217.65:33967: connect: Connection refused (error 111)
I20260926 07:04:07.422430 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14957
I20260926 07:04:07.430289 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 14589
I20260926 07:04:07.437158 1893 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 07:04:07.782147 1893 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 07:04:07.782240 1893 mini_ranger.cc:66] Stopping Ranger...
I20260926 07:04:08.127703 1893 mini_ranger.cc:68] Stopped Ranger
2026-09-26 07:04:08.127 UTC [13656] LOG: received smart shutdown request
2026-09-26 07:04:08.132 UTC [13656] LOG: background worker "logical replication launcher" (PID 13665) exited with exit code 1
2026-09-26 07:04:08.132 UTC [13660] LOG: shutting down
2026-09-26 07:04:08.296 UTC [13656] LOG: database system is shut down
2026-09-26T07:04:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47629 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:04:08 dist-test-slave-t1mc krb5kdc[15159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:04:08 dist-test-slave-t1mc krb5kdc[15159](info): set up 2 sockets
Sep 26 07:04:08 dist-test-slave-t1mc krb5kdc[15159](info): commencing operation
krb5kdc: starting...
W20260926 07:04:09.428706 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:04:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:04:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:04:09.545940 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:44209
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:09.627825 15172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:09.628084 15172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:09.628123 15172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:09.629357 15172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:09.629431 15172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:09.629455 15172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:09.629475 15172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:09.629508 15172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:09.631449 15172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:44209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:09.631773 15172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:09.632009 15172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:09.636828 15177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:09.636929 15178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:09.637027 15172 server_base.cc:1034] running on GCE node
W20260926 07:04:09.637032 15180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:09.637358 15172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:09.637660 15172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:09.638823 15172 hybrid_clock.cc:648] HybridClock initialized: now 1790406249638809 us; error 30 us; skew 500 ppm
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:09.641393 15172 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:09.641825 15172 webserver.cc:460] Webserver started at http://127.1.217.126:37583/ using document root <none> and password file <none>
I20260926 07:04:09.642051 15172 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:09.642118 15172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:09.642264 15172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:09.643349 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "0f084a49212d4c45900adb554d318848"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "505547103ce37f8cd5e88ad9c2127a55"
server_key_iv: "161a8c5fe125f1e70af8d6b7a9ed530d"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.643712 15172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "0f084a49212d4c45900adb554d318848"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "505547103ce37f8cd5e88ad9c2127a55"
server_key_iv: "161a8c5fe125f1e70af8d6b7a9ed530d"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.645102 15172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:04:09.645839 15187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:09.646014 15172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:09.646117 15172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "0f084a49212d4c45900adb554d318848"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "505547103ce37f8cd5e88ad9c2127a55"
server_key_iv: "161a8c5fe125f1e70af8d6b7a9ed530d"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.646189 15172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:09.666090 15172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:09.668344 15172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:09.668516 15172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:09.672441 15172 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:44209
I20260926 07:04:09.672508 15239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:44209 every 8 connection(s)
I20260926 07:04:09.672886 15172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:09.673718 15240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:09.675194 15240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:09.680998 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15172
I20260926 07:04:09.681113 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:04:09.681289 1893 external_mini_cluster.cc:1428] Setting key 7a7f6d3a16c955a6ffc2a0f3e838507f
I20260926 07:04:09.684471 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.675833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44209 (local address 127.0.0.1:51376)
0926 07:04:09.675987 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.676025 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.676476 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.676729 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.676731 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.677031 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.677190 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.677194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.678163 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.678164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.678843 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.678844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.679294 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.679295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.679358 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.679811 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.679819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.681280 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.683048 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.683054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.683056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.683248 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.683477 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.683477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.683478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.683566 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.683900 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.683904 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.683964 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.684072 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.684285 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":81,"thread_start_us":81,"threads_started":1}
I20260926 07:04:09.684476 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.675871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.1.217.126:44209)
0926 07:04:09.676023 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.676039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.676100 (+ 61us) server_negotiation.cc:407] Connection header received
0926 07:04:09.676600 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.676602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.676618 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.676701 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.677275 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.678080 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.678915 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.679221 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.679299 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.681381 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.681388 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.681388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.681400 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.682950 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.683330 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.683334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.683334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.683411 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.683631 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.683632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.683632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.683834 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.683920 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.684096 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.684225 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 07:04:09.685171 15240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.685529 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.685400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60552)
0926 07:04:09.685449 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.685505 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:04:09.685658 15188 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
I20260926 07:04:09.686065 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.685976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60560)
0926 07:04:09.686021 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.686044 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:04:09.686162 15240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34413: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.217.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
I20260926 07:04:09.707013 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34413
--webserver_interface=127.1.217.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:09.735417 15240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } attempt: 1
I20260926 07:04:09.735975 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.735819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60564)
0926 07:04:09.735887 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.735936 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:04:09.736444 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.736321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60568)
0926 07:04:09.736380 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.736418 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 07:04:09.736578 15240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34413: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
W20260926 07:04:09.788058 15247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:09.788303 15247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:09.788341 15247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:09.789528 15247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:09.789597 15247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:09.789623 15247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:09.789651 15247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:09.789695 15247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:09.791234 15247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.217.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:09.791481 15247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:09.791652 15247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:09.796415 15253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:09.796444 15252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:09.796415 15255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:09.796614 15247 server_base.cc:1034] running on GCE node
I20260926 07:04:09.796947 15247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:09.797238 15247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:09.798416 15247 hybrid_clock.cc:648] HybridClock initialized: now 1790406249798402 us; error 42 us; skew 500 ppm
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:09.800911 15247 init.cc:377] Logged in from keytab as kudu/127.1.217.125@KRBTEST.COM (short username kudu)
I20260926 07:04:09.801380 15247 webserver.cc:460] Webserver started at http://127.1.217.125:35701/ using document root <none> and password file <none>
I20260926 07:04:09.801612 15247 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:09.801681 15247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:09.801838 15247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:09.802970 15247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data/instance:
uuid: "2d025e9554cd405f92615a07519dcd4e"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "cf98dff6e44785f42b747dfd8c4563df"
server_key_iv: "ebaebb5aaace2e81c1a0248f39b512fc"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.803339 15247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal/instance:
uuid: "2d025e9554cd405f92615a07519dcd4e"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "cf98dff6e44785f42b747dfd8c4563df"
server_key_iv: "ebaebb5aaace2e81c1a0248f39b512fc"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.804747 15240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } attempt: 2
I20260926 07:04:09.804868 15247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:04:09.805145 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.805030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60584)
0926 07:04:09.805079 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.805116 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 07:04:09.805526 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.805418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60590)
0926 07:04:09.805457 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.805501 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 07:04:09.805639 15240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34413: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34413: connect: Connection refused (error 111)
I20260926 07:04:09.805773 15262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:09.805919 15247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:09.806025 15247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "2d025e9554cd405f92615a07519dcd4e"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "cf98dff6e44785f42b747dfd8c4563df"
server_key_iv: "ebaebb5aaace2e81c1a0248f39b512fc"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.806100 15247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:09.814733 15247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:09.816519 15247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:09.816682 15247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:09.820577 15247 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.125:34413
I20260926 07:04:09.820613 15314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.125:34413 every 8 connection(s)
I20260926 07:04:09.821043 15247 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
I20260926 07:04:09.821796 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15247
I20260926 07:04:09.821918 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-1/wal/instance
I20260926 07:04:09.821950 15315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:09.822183 1893 external_mini_cluster.cc:1428] Setting key e5b2f5dcce6dafde015e57d7a66f49f5
I20260926 07:04:09.823864 15315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.217.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.124@KRBTEST.COM" created.
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:09.836836 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.824709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51380 (local address 127.1.217.126:44209)
0926 07:04:09.824797 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.824799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.825001 (+ 202us) server_negotiation.cc:407] Connection header received
0926 07:04:09.825743 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.825744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.825757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.825801 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.826641 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.827119 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.828670 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.828918 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.828972 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.833672 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.833674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.833674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.833683 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.835389 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.835810 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.835811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.835811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.835864 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.836155 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.836156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.836157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.836279 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.836352 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.836626 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.836713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:04:09.837051 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.824680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44209 (local address 127.0.0.1:51380)
0926 07:04:09.824864 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.824912 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.825633 (+ 721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.825878 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.825882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.826245 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.826551 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.826558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.827211 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.827212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.828588 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.828592 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.829000 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.829001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.829158 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.829653 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.829664 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.833567 (+ 3903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.835477 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.835485 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.835487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.835743 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.835976 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.835977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.835978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.836087 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.836355 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.836362 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.836497 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.836606 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.836771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":114,"threads_started":1}
I20260926 07:04:09.837599 15315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:04:09.846532 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.837888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60604)
0926 07:04:09.837942 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.837954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.837994 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.838231 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.838232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.838410 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.838484 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.838485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.839316 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.839317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.840180 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.840183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.840633 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.840634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.840722 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.841116 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.841121 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.843982 (+ 2861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.845619 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.845620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.845621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.845793 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.845965 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.845966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.845966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.846015 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.846282 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.846283 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.846317 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.846367 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.846445 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:04:09.846597 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.837963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.1.217.125:34413)
0926 07:04:09.838103 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.838107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.838121 (+ 14us) server_negotiation.cc:407] Connection header received
0926 07:04:09.838135 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.838137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.838146 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.838196 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.838565 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.839226 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.840262 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.840553 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.840604 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.844062 (+ 3458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.844070 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.844070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.844077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.845550 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.845856 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.845858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.845859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.845909 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.846068 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.846069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.846069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.846226 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.846293 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.846390 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.846492 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260926 07:04:09.847200 15315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.847533 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.847452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58338)
0926 07:04:09.847478 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.847517 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 07:04:09.847664 15265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
I20260926 07:04:09.848125 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.848030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58348)
0926 07:04:09.848078 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.848109 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:04:09.848233 15315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:36355: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
I20260926 07:04:09.849535 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:36355
--webserver_interface=127.1.217.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:39471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:09.896822 15240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } attempt: 3
I20260926 07:04:09.897451 15315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } attempt: 1
I20260926 07:04:09.897868 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.897717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58358)
0926 07:04:09.897779 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.897821 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:04:09.898207 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.898110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58370)
0926 07:04:09.898155 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.898181 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:04:09.898309 15315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:36355: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:04:09.904577 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.897177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60614)
0926 07:04:09.897245 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.897259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.897339 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.897539 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.897539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.897723 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.897819 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.897821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.898833 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.898834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.899371 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.899373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.899845 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.899846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.899909 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.900373 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.900377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.901831 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.903509 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.903510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.903510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.903692 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.903893 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.903894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.903894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.903957 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.904207 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.904208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.904253 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.904366 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.904469 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:04:09.904675 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.897253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.1.217.125:34413)
0926 07:04:09.897326 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.897329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.897341 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:04:09.897451 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.897455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.897468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.897534 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.897903 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.898704 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.899451 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.899766 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.899834 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.901912 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.901913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.901914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.901927 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.903423 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.903764 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.903765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.903766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.903827 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.904015 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.904015 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.904016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.904144 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.904212 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.904487 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.904565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 07:04:09.905043 15240 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.905370 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.905277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58380)
0926 07:04:09.905325 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.905353 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 07:04:09.905647 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.905597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58384)
0926 07:04:09.905617 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.905635 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 07:04:09.905738 15240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:36355: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
W20260926 07:04:09.930787 15322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:09.931001 15322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:09.931042 15322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:09.932176 15322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:09.932233 15322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:09.932258 15322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:09.932286 15322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:09.932313 15322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:09.933831 15322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:39471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44209,127.1.217.125:34413,127.1.217.124:36355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:36355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.217.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:09.934092 15322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:09.934302 15322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:04:09.936877 15240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } attempt: 1
I20260926 07:04:09.937343 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.937188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58400)
0926 07:04:09.937266 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.937317 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 07:04:09.937669 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.937568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58408)
0926 07:04:09.937613 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.937638 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:04:09.937849 15240 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:36355: Network error: Client connection negotiation failed: client connection to 127.1.217.124:36355: connect: Connection refused (error 111)
W20260926 07:04:09.938802 15330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:09.938817 15327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:09.938800 15328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:09.939146 15322 server_base.cc:1034] running on GCE node
I20260926 07:04:09.939424 15322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:09.939705 15322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:09.940843 15322 hybrid_clock.cc:648] HybridClock initialized: now 1790406249940827 us; error 29 us; skew 500 ppm
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:09.943174 15322 init.cc:377] Logged in from keytab as kudu/127.1.217.124@KRBTEST.COM (short username kudu)
I20260926 07:04:09.943616 15322 webserver.cc:460] Webserver started at http://127.1.217.124:33207/ using document root <none> and password file <none>
I20260926 07:04:09.943837 15322 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:09.943877 15322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:09.943969 15322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:09.945051 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data/instance:
uuid: "a061924ba75641d9a715f13eb03ec121"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "f27249d9cca69b01faa548a9eb016520"
server_key_iv: "6ddcd9beec17756ac37aaf2b69a7aba7"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.945348 15322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal/instance:
uuid: "a061924ba75641d9a715f13eb03ec121"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "f27249d9cca69b01faa548a9eb016520"
server_key_iv: "6ddcd9beec17756ac37aaf2b69a7aba7"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.946545 15322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:09.947172 15337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:09.947362 15322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:09.947423 15322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "a061924ba75641d9a715f13eb03ec121"
format_stamp: "Formatted at 2026-09-26 07:04:09 on dist-test-slave-t1mc"
server_key: "f27249d9cca69b01faa548a9eb016520"
server_key_iv: "6ddcd9beec17756ac37aaf2b69a7aba7"
server_key_version: "encryptionkey@0"
I20260926 07:04:09.947479 15322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:09.953667 15322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:09.955288 15322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:09.955391 15322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:09.959182 15322 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.124:36355
I20260926 07:04:09.959235 15389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.124:36355 every 8 connection(s)
I20260926 07:04:09.959564 15322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
I20260926 07:04:09.960341 15390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:09.961862 15390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.963786 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15322
I20260926 07:04:09.963896 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790406101908495-1893-0/minicluster-data/master-2/wal/instance
I20260926 07:04:09.964104 1893 external_mini_cluster.cc:1428] Setting key d85863f3e68cb12bd08f6283c12b4f0a
I20260926 07:04:09.966491 15315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } attempt: 2
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:04:09.971638 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.962505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51388 (local address 127.1.217.126:44209)
0926 07:04:09.962594 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.962596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.962743 (+ 147us) server_negotiation.cc:407] Connection header received
0926 07:04:09.963229 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.963230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.963242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.963319 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.964100 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.964683 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.965848 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.966061 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.966115 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.968379 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.968394 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.968394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.968406 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.970063 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.970494 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.970495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.970495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.970545 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.970854 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.970855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.970856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.971013 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.971098 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.971444 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.971527 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 07:04:09.971825 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.962440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44209 (local address 127.0.0.1:51388)
0926 07:04:09.962633 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.962666 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.963126 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.963387 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.963391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.963676 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.964006 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.964012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.964790 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.964791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.965769 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.965771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.966134 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.966135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.966275 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.966823 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.966832 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.968279 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.970152 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.970158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.970160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.970385 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.970620 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.970621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.970621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.970738 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.971089 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.971096 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.971283 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.971418 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.971589 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:04:09.972956 15390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.974184 15394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.964488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.1.217.124:36355)
0926 07:04:09.964823 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.964827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.964847 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:04:09.964866 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.964868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.964887 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.964962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.965522 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.966489 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.967208 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.967602 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.967682 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.970438 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.970452 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.970453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.970459 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.972388 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.972823 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.972825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.972825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.972888 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.973334 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.973335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.973336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.973515 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.973629 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.974028 (+ 399us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.974117 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":90,"threads_started":1}
I20260926 07:04:09.976446 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.968181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.1.217.124:36355)
0926 07:04:09.968331 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.968333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.968342 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:09.968361 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.968362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.968373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.968468 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.968908 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.969642 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.970589 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.970925 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.970998 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.972836 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.972838 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.972838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.972851 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.975267 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.975641 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.975642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.975642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.975694 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.975916 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.975917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.975917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.976045 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.976113 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.976256 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.976327 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260926 07:04:09.976451 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.967335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58420)
0926 07:04:09.967401 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.967413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.967468 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.968509 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.968510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.968684 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.968728 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.968730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.969735 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.969736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.970512 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.970514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.971001 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.971003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.971062 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.971509 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.971514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.972723 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.975360 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.975361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.975362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.975563 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.975765 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.975766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.975766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.975834 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.976112 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.976113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.976156 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.976230 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.976326 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:04:09.978374 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: Bootstrap starting.
I20260926 07:04:09.979336 15315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: Neither blocks nor log segments found. Creating new log.
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:09.979630 15315 log.cc:824] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:09.980464 15315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: No bootstrap required, opened a new log
I20260926 07:04:09.981324 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.973187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34413 (local address 127.0.0.1:60624)
0926 07:04:09.973262 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.973277 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.973334 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.973453 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.973454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.973641 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.973697 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.973699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.974599 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.974600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.975519 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.975521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.976009 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.976009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.976078 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.976577 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.976582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.977905 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.980185 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.980186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.980186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.980394 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.980623 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.980624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.980624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.980681 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.981002 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.981003 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.981045 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.981144 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.981233 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 07:04:09.981323 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.973242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.1.217.125:34413)
0926 07:04:09.973279 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.973281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.973321 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:04:09.973382 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.973383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.973392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.973449 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.973757 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.974480 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.975611 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.975899 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.975940 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.977987 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.977988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.977988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.977995 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.980119 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.980491 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.980492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.980493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.980546 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.980739 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.980740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.980740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.980942 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.981009 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.981172 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.981241 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:09.982013 15390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:04:09.982863 15315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:09.983080 15315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:09.983168 15315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d025e9554cd405f92615a07519dcd4e, State: Initialized, Role: FOLLOWER
I20260926 07:04:09.983210 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.974921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.1.217.126:44209)
0926 07:04:09.974980 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.974982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.974990 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:09.975052 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.975053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.975061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.975105 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.975548 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.976128 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.977217 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.977425 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.977509 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.979889 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.979890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.979890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.979896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.981624 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.982025 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.982026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.982026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.982086 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.982352 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.982353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.982354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.982480 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.982994 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.983039 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.983112 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:04:09.983368 15315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:09.983749 15315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:09.983713 15397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:09.985168 15397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
W20260926 07:04:09.986610 15408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:04:09.986662 15408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 07:04:09 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:04:09.989319 15392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.982280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58426)
0926 07:04:09.982318 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:09.982326 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:09.982361 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:09.982489 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:09.982489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:09.982616 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:09.982639 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.982641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.983346 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.983347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.984133 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:09.984135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.984640 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.984641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:09.984695 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.985122 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:09.985126 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:09.986460 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:09.988322 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.988323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.988323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.988524 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.988701 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:09.988701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:09.988701 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:09.988746 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:09.989059 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:09.989060 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:09.989092 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:04:09.989133 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 07:04:09.989217 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 07:04:09.989651 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.982318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.1.217.124:36355)
0926 07:04:09.982352 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.982353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.982361 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:09.982424 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.982424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.982433 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.982486 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.982720 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.983243 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.984223 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.984574 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.984645 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.986549 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.986551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.986551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.986559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.988232 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.988595 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.988596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.988596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.988640 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.988829 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.988829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.988830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.988986 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.989433 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.989494 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.989580 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:04:09.987752 15247 master_runner.cc:186] Error getting master registration for 127.1.217.126:44209: OK, instance_id { permanent_uuid: "0f084a49212d4c45900adb554d318848" instance_seqno: 1790406249671374 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:04:09.990569 15247 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:04:09.991806 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.983758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.1.217.125:34413)
0926 07:04:09.983904 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.983905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.983912 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:04:09.983929 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.983929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.983938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.984011 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.984596 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.985296 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.986068 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.986350 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.986420 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:09.989111 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:09.989112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:09.989113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:09.989120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:09.990777 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.991149 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.991150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.991150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.991199 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:09.991389 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:09.991390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:09.991390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:09.991479 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:09.991569 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:09.991687 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:04:09.991754 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260926 07:04:09.992110 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: Bootstrap starting.
I20260926 07:04:09.992921 15390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:09.993281 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:04:09.993273 15390 log.cc:824] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:09.994160 15390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: No bootstrap required, opened a new log
I20260926 07:04:09.996361 15390 raft_consensus.cc:348] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:09.996690 15390 raft_consensus.cc:374] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:09.996812 15390 raft_consensus.cc:729] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a061924ba75641d9a715f13eb03ec121, State: Initialized, Role: FOLLOWER
I20260926 07:04:09.997009 15390 consensus_queue.cc:260] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:09.997372 15390 sys_catalog.cc:564] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 07:04:10.002502 15429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:04:10.002600 15429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:04:09.999437 15322 master_runner.cc:186] Error getting master registration for 127.1.217.126:44209: OK, instance_id { permanent_uuid: "0f084a49212d4c45900adb554d318848" instance_seqno: 1790406249671374 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:04:10.003196 15322 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:04:10.003053 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.995307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58438 (local address 127.1.217.124:36355)
0926 07:04:09.995346 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.995348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.995635 (+ 287us) server_negotiation.cc:407] Connection header received
0926 07:04:09.995724 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.995725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.995734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.995761 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.996440 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.997156 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.998160 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.998464 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.998505 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:10.000069 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:10.000071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:10.000071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:10.000078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:10.001729 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.002181 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.002182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.002183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.002237 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.002448 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.002449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.002450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.002567 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:10.002673 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:10.002880 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 07:04:10.002958 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:10.004045 15418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.006541 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.995348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.1.217.125:34413)
0926 07:04:09.996571 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.996573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.996581 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:09.996599 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.996600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.996608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.996660 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.999419 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.000128 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.001149 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.001428 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.001468 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:10.003625 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:10.003627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:10.003627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:10.003634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:10.005307 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.005683 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.005684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.005685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.005738 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.005941 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.005942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.005942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.006082 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:10.006166 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:10.006376 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 07:04:10.006450 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202}
I20260926 07:04:10.007395 15418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:04:10.007438 15243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:09.995268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.1.217.126:44209)
0926 07:04:09.995391 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:09.995393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:09.995402 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:09.995435 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:09.995435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:09.995445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:09.995510 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:09.996199 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.996919 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.997967 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:09.998265 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:09.998308 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:10.004525 (+ 6217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:10.004526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:10.004527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:10.004535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:10.006222 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.006625 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.006626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.006627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.006680 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.006867 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.006868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.006868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.006987 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:10.007052 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:10.007255 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 07:04:10.007333 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 07:04:10.012985 15240 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } attempt: 2
Sep 26 07:04:10 dist-test-slave-t1mc krb5kdc[15159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406249, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:04:10.020923 15395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:10.013278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.1.217.124:36355)
0926 07:04:10.013318 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:10.013321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:10.013329 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:10.013371 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:10.013372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:10.013381 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:10.013443 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:10.013749 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.014481 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.015426 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.015710 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.015779 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:10.017951 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:10.017952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:10.017953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:10.017966 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:10.019657 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.020044 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.020045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.020046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.020103 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:10.020308 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:10.020308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:10.020309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:10.020471 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:10.020538 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:10.020703 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:04:10.020826 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:04:10.020934 15242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:10.013227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:36355 (local address 127.0.0.1:58454)
0926 07:04:10.013270 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:10.013281 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:10.013317 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:10.013463 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:10.013464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:10.013625 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:10.013669 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:10.013671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.014567 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.014568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:10.015355 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:10.015357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:10.015788 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:10.015789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:10.015848 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:10.016326 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:10.016331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:10.017863 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:10.019741 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:10.019742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:10.019743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:10.019966 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:10.020174 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:10.020175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:10.020175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:10.020240 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:10.020543 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:10.020544 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:10.020593 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 07:04:10.020668 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:04:10.020828 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:04:10.022859 15240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: Bootstrap starting.
I20260926 07:04:10.023589 15240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:10.023870 15240 log.cc:824] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:10.024734 15240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: No bootstrap required, opened a new log
I20260926 07:04:10.026808 15240 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:10.026953 15240 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:10.026989 15240 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f084a49212d4c45900adb554d318848, State: Initialized, Role: FOLLOWER
I20260926 07:04:10.027133 15240 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:10.027403 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.027467 15240 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:10.027554 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:04:10.030665 15443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:04:10.030721 15443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:04:10.031591 15172 master_runner.cc:263] No leader master found from master 0f084a49212d4c45900adb554d318848
I20260926 07:04:10.031637 15172 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:04:10.067354 15268 tablet.cc:2369] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:10.153594 15397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:10.153726 15397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:10.153985 15397 leader_election.cc:288] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f084a49212d4c45900adb554d318848 (127.1.217.126:44209), a061924ba75641d9a715f13eb03ec121 (127.1.217.124:36355)
I20260926 07:04:10.154277 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d025e9554cd405f92615a07519dcd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f084a49212d4c45900adb554d318848" is_pre_election: true
I20260926 07:04:10.154335 15364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d025e9554cd405f92615a07519dcd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a061924ba75641d9a715f13eb03ec121" is_pre_election: true
I20260926 07:04:10.154438 15214 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d025e9554cd405f92615a07519dcd4e in term 0.
I20260926 07:04:10.154461 15364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d025e9554cd405f92615a07519dcd4e in term 0.
I20260926 07:04:10.154633 15265 leader_election.cc:302] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d025e9554cd405f92615a07519dcd4e, a061924ba75641d9a715f13eb03ec121; no voters:
I20260926 07:04:10.154762 15397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:10.154804 15397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:10.154830 15397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:10.155552 15397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:10.155691 15397 leader_election.cc:288] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [CANDIDATE]: Term 1 election: Requested vote from peers 0f084a49212d4c45900adb554d318848 (127.1.217.126:44209), a061924ba75641d9a715f13eb03ec121 (127.1.217.124:36355)
I20260926 07:04:10.155853 15364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d025e9554cd405f92615a07519dcd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a061924ba75641d9a715f13eb03ec121"
I20260926 07:04:10.155862 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d025e9554cd405f92615a07519dcd4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f084a49212d4c45900adb554d318848"
I20260926 07:04:10.155928 15214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:10.155928 15364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:10.156736 15214 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d025e9554cd405f92615a07519dcd4e in term 1.
I20260926 07:04:10.156739 15364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d025e9554cd405f92615a07519dcd4e in term 1.
I20260926 07:04:10.156900 15265 leader_election.cc:302] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d025e9554cd405f92615a07519dcd4e, a061924ba75641d9a715f13eb03ec121; no voters:
I20260926 07:04:10.156972 15397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:10.157177 15397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [term 1 LEADER]: Becoming Leader. State: Replica: 2d025e9554cd405f92615a07519dcd4e, State: Running, Role: LEADER
I20260926 07:04:10.157272 15397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } }
I20260926 07:04:10.157652 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d025e9554cd405f92615a07519dcd4e. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.157790 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:10.158023 15447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:10.158211 15447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:10.159482 15214 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [term 1 FOLLOWER]: Refusing update from remote peer 2d025e9554cd405f92615a07519dcd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:10.159492 15364 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [term 1 FOLLOWER]: Refusing update from remote peer 2d025e9554cd405f92615a07519dcd4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:10.159761 15445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:10.159840 15397 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:10.160969 15418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d025e9554cd405f92615a07519dcd4e. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.160969 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d025e9554cd405f92615a07519dcd4e. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.161051 15418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:04:10.161055 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:04:10.161847 15448 mvcc.cc:205] Tried to move back new op lower bound from 7333504000650915840 to 7333504000644128768. Current Snapshot: MvccSnapshot[applied={T|T < 7333504000650915840}]
I20260926 07:04:10.162199 15418 sys_catalog.cc:455] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.162276 15418 sys_catalog.cc:458] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:04:10.162761 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.162842 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:04:10.163345 15445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.163722 15450 mvcc.cc:205] Tried to move back new op lower bound from 7333504000650915840 to 7333504000644128768. Current Snapshot: MvccSnapshot[applied={T|T < 7333504000650915840}]
I20260926 07:04:10.163514 15397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d025e9554cd405f92615a07519dcd4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f084a49212d4c45900adb554d318848" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44209 } } peers { permanent_uuid: "2d025e9554cd405f92615a07519dcd4e" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34413 } } peers { permanent_uuid: "a061924ba75641d9a715f13eb03ec121" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 36355 } } }
I20260926 07:04:10.164141 15445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:10.164150 15449 mvcc.cc:205] Tried to move back new op lower bound from 7333504000650915840 to 7333504000644128768. Current Snapshot: MvccSnapshot[applied={T|T < 7333504000650915840}]
I20260926 07:04:10.164196 15397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:10.165151 15447 catalog_manager.cc:1347] Generated new cluster ID: 71c0a75d98dd45e1a0b806b587e0257c
I20260926 07:04:10.165218 15447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:04:10.178261 15447 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:04:10.179064 15447 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:10.187198 15447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d025e9554cd405f92615a07519dcd4e: Generated new TSK 0
I20260926 07:04:10.187379 15447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:04:11.003089 15429 catalog_manager.cc:1259] Loaded cluster ID: 71c0a75d98dd45e1a0b806b587e0257c
I20260926 07:04:11.003168 15429 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: loading cluster ID for follower catalog manager: success
I20260926 07:04:11.003997 15429 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: acquiring CA information for follower catalog manager: success
I20260926 07:04:11.004319 15429 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a061924ba75641d9a715f13eb03ec121: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:04:11.031224 15443 catalog_manager.cc:1259] Loaded cluster ID: 71c0a75d98dd45e1a0b806b587e0257c
I20260926 07:04:11.031332 15443 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: loading cluster ID for follower catalog manager: success
I20260926 07:04:11.032168 15443 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: acquiring CA information for follower catalog manager: success
I20260926 07:04:11.032485 15443 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0f084a49212d4c45900adb554d318848: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:04:11.247675 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15172
I20260926 07:04:11.253033 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15247
I20260926 07:04:11.257851 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15322
2026-09-26T07:04:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2855 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:04:11 dist-test-slave-t1mc krb5kdc[15463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:04:11 dist-test-slave-t1mc krb5kdc[15463](info): set up 2 sockets
Sep 26 07:04:11 dist-test-slave-t1mc krb5kdc[15463](info): commencing operation
krb5kdc: starting...
W20260926 07:04:12.300753 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:04:12.435395 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43165
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:12.514693 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:12.514941 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:12.514982 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:12.516170 15476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:12.516235 15476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:12.516260 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:12.516280 15476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:12.516309 15476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:12.517853 15476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43165
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:12.518123 15476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:12.518348 15476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:12.523125 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:12.523244 15481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:12.523375 15482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:12.523840 15476 server_base.cc:1034] running on GCE node
I20260926 07:04:12.524084 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:12.524364 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:12.525524 15476 hybrid_clock.cc:648] HybridClock initialized: now 1790406252525513 us; error 45 us; skew 500 ppm
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406252, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:12.527976 15476 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:12.528446 15476 webserver.cc:460] Webserver started at http://127.1.217.126:34405/ using document root <none> and password file <none>
I20260926 07:04:12.528683 15476 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:12.528750 15476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:12.528923 15476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:12.529964 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "a6aa60e073634504baea9aa8c06528a3"
format_stamp: "Formatted at 2026-09-26 07:04:12 on dist-test-slave-t1mc"
server_key: "b9a820192d324097142893bfe94557ca"
server_key_iv: "cf086abc46c21a65a18f72b454b72dd2"
server_key_version: "encryptionkey@0"
I20260926 07:04:12.530331 15476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "a6aa60e073634504baea9aa8c06528a3"
format_stamp: "Formatted at 2026-09-26 07:04:12 on dist-test-slave-t1mc"
server_key: "b9a820192d324097142893bfe94557ca"
server_key_iv: "cf086abc46c21a65a18f72b454b72dd2"
server_key_version: "encryptionkey@0"
I20260926 07:04:12.531623 15476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:04:12.532387 15491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:12.532648 15476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:04:12.532747 15476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "a6aa60e073634504baea9aa8c06528a3"
format_stamp: "Formatted at 2026-09-26 07:04:12 on dist-test-slave-t1mc"
server_key: "b9a820192d324097142893bfe94557ca"
server_key_iv: "cf086abc46c21a65a18f72b454b72dd2"
server_key_version: "encryptionkey@0"
I20260926 07:04:12.532856 15476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:12.553100 15476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:12.554973 15476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:12.555136 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:12.559284 15476 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43165
I20260926 07:04:12.559317 15543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43165 every 8 connection(s)
I20260926 07:04:12.559666 15476 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:12.559978 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15476
I20260926 07:04:12.560086 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:04:12.560237 1893 external_mini_cluster.cc:1428] Setting key 93820a3307186abd3e02b995c36f7de0
I20260926 07:04:12.560686 15544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:12.563942 15544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: Bootstrap starting.
I20260926 07:04:12.564576 15544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:12.564908 15544 log.cc:824] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:12.565593 15544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: No bootstrap required, opened a new log
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:12.568003 15544 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } }
I20260926 07:04:12.568208 15544 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:12.568250 15544 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6aa60e073634504baea9aa8c06528a3, State: Initialized, Role: FOLLOWER
I20260926 07:04:12.568447 15544 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } }
I20260926 07:04:12.568534 15544 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:12.568558 15544 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:12.568607 15544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:12.569231 15544 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } }
I20260926 07:04:12.569325 15544 leader_election.cc:302] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6aa60e073634504baea9aa8c06528a3; no voters:
I20260926 07:04:12.569516 15544 leader_election.cc:288] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:04:12.569571 15549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:12.569758 15549 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [term 1 LEADER]: Becoming Leader. State: Replica: a6aa60e073634504baea9aa8c06528a3, State: Running, Role: LEADER
I20260926 07:04:12.569921 15549 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } }
I20260926 07:04:12.569792 15544 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:12.570259 15550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6aa60e073634504baea9aa8c06528a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } } }
I20260926 07:04:12.570439 15551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6aa60e073634504baea9aa8c06528a3. Latest consensus state: current_term: 1 leader_uuid: "a6aa60e073634504baea9aa8c06528a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6aa60e073634504baea9aa8c06528a3" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43165 } } }
I20260926 07:04:12.570350 15550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:12.570609 15551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:12.571925 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:12.560691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.1.217.126:43165)
0926 07:04:12.560886 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:12.560895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:12.560926 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:04:12.561459 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:12.561468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:12.561531 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:12.561687 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:12.562255 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:12.563640 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:12.564429 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:12.565101 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:12.565251 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:12.567796 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:12.567800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:12.567800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:12.567810 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:12.569763 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:12.570215 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:12.570218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:12.570219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:12.570320 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:12.570555 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:12.570556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:12.570556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:12.570864 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:12.571399 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:12.571474 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 07:04:12.571637 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
W20260926 07:04:12.572197 15565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:04:12.572261 15565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:04:12.572328 15558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:12.572465 15558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:12.572824 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:04:12.574008 15558 catalog_manager.cc:1347] Generated new cluster ID: a6f6dc9cb0ef48cfaaa2e3ba7b9e1ebb
I20260926 07:04:12.574059 15558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:04:12.577952 15558 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:04:12.578610 15558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:12.583765 15558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6aa60e073634504baea9aa8c06528a3: Generated new TSK 0
I20260926 07:04:12.583931 15558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:04:12.626760 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15476
2026-09-26T07:04:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1367 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15571](info): set up 2 sockets
Sep 26 07:04:12 dist-test-slave-t1mc krb5kdc[15571](info): commencing operation
krb5kdc: starting...
W20260926 07:04:13.668915 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:04:13 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:04:13.783835 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:13.863876 15584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:13.864140 15584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:13.864184 15584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:13.865370 15584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:13.865443 15584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:13.865466 15584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:13.865487 15584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:13.865546 15584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:13.867420 15584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:13.867713 15584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:13.867954 15584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:13.872902 15592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:13.872849 15590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:13.872853 15589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:13.873270 15584 server_base.cc:1034] running on GCE node
I20260926 07:04:13.873533 15584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:13.873862 15584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:13.875033 15584 hybrid_clock.cc:648] HybridClock initialized: now 1790406253875014 us; error 48 us; skew 500 ppm
Sep 26 07:04:13 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406253, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:13.877552 15584 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:13.878048 15584 webserver.cc:460] Webserver started at http://127.1.217.126:42529/ using document root <none> and password file <none>
I20260926 07:04:13.878293 15584 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:13.878361 15584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:13.878513 15584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:13.879575 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d6ff1e6e7d18422c8e2f14472ef2d001"
format_stamp: "Formatted at 2026-09-26 07:04:13 on dist-test-slave-t1mc"
server_key: "36f1eee20370e27ff6e1337b758ab934"
server_key_iv: "bea387760e246348659681f22867705b"
server_key_version: "encryptionkey@0"
I20260926 07:04:13.879941 15584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d6ff1e6e7d18422c8e2f14472ef2d001"
format_stamp: "Formatted at 2026-09-26 07:04:13 on dist-test-slave-t1mc"
server_key: "36f1eee20370e27ff6e1337b758ab934"
server_key_iv: "bea387760e246348659681f22867705b"
server_key_version: "encryptionkey@0"
I20260926 07:04:13.881345 15584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:04:13.882205 15599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:13.882407 15584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:13.882516 15584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d6ff1e6e7d18422c8e2f14472ef2d001"
format_stamp: "Formatted at 2026-09-26 07:04:13 on dist-test-slave-t1mc"
server_key: "36f1eee20370e27ff6e1337b758ab934"
server_key_iv: "bea387760e246348659681f22867705b"
server_key_version: "encryptionkey@0"
I20260926 07:04:13.882592 15584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:13.915786 15584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:13.917780 15584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:13.917948 15584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:13.922052 15584 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43825
I20260926 07:04:13.922109 15651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43825 every 8 connection(s)
I20260926 07:04:13.922479 15584 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:13.923509 15652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:13.925819 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap starting.
I20260926 07:04:13.926353 15652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:13.926582 15652 log.cc:824] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:13.927227 15652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: No bootstrap required, opened a new log
I20260926 07:04:13.928848 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15584
I20260926 07:04:13.928967 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:04:13.928871 15652 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:13.929064 15652 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:13.929111 15652 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Initialized, Role: FOLLOWER
I20260926 07:04:13.929167 1893 external_mini_cluster.cc:1428] Setting key 1cdbc4c8295ac855dccb19515fa0931e
I20260926 07:04:13.929253 15652 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:13.929376 15652 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:13.929426 15652 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:13.929486 15652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:13.930203 15652 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:13.930327 15652 leader_election.cc:302] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ff1e6e7d18422c8e2f14472ef2d001; no voters:
I20260926 07:04:13.930491 15652 leader_election.cc:288] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:04:13.930512 15657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:13.930651 15657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 LEADER]: Becoming Leader. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Running, Role: LEADER
I20260926 07:04:13.930744 15652 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:13.930797 15657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:13.931191 15658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:13.931301 15658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:13.931191 15657 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ff1e6e7d18422c8e2f14472ef2d001. Latest consensus state: current_term: 1 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:13.931494 15657 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:13.931689 15666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:13.932225 15666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:13.933890 15666 catalog_manager.cc:1347] Generated new cluster ID: 21ed797489914c0fabece88bb27981a6
I20260926 07:04:13.933941 15666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:04:13 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:13.940820 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:13.929619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.1.217.126:43825)
0926 07:04:13.929783 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:13.929793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:13.929826 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:04:13.930145 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:13.930150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:13.930208 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:13.930364 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:13.931158 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:13.932582 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:13.933706 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:13.934286 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:13.934388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:13.937110 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:13.937114 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:13.937115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:13.937126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:13.939099 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:13.939483 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:13.939487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:13.939487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:13.939629 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:13.939843 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:13.939843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:13.939844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:13.940217 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:13.940372 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:13.940443 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:04:13.940588 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 07:04:13.955422 15666 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:04:13.955935 15666 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:13.959679 15666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Generated new TSK 0
I20260926 07:04:13.959868 15666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:04:13.987152 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:14.067576 15678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.067819 15678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.067855 15678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.069058 15678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.069116 15678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.069161 15678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:14.070606 15678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.070890 15678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.071122 15678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.076189 15684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.076155 15683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.076323 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.076383 15678 server_base.cc:1034] running on GCE node
I20260926 07:04:14.076588 15678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.076898 15678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.078084 15678 hybrid_clock.cc:648] HybridClock initialized: now 1790406254078059 us; error 37 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.080482 15678 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:14.080976 15678 webserver.cc:460] Webserver started at http://127.1.217.65:45741/ using document root <none> and password file <none>
I20260926 07:04:14.081214 15678 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.081279 15678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.081410 15678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:14.082476 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "7b9c26daf6a84586a83a1a9e7600e788"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "deb4f20af3ad951b1e5c5beffded4f4d"
server_key_iv: "21333fc35bfca45360b50f6fb41575c3"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.082840 15678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "7b9c26daf6a84586a83a1a9e7600e788"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "deb4f20af3ad951b1e5c5beffded4f4d"
server_key_iv: "21333fc35bfca45360b50f6fb41575c3"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.084108 15678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:04:14.084844 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.085017 15678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:14.085116 15678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "7b9c26daf6a84586a83a1a9e7600e788"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "deb4f20af3ad951b1e5c5beffded4f4d"
server_key_iv: "21333fc35bfca45360b50f6fb41575c3"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.085201 15678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.095582 15678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.097437 15678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.097604 15678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.097828 15678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:14.098140 15678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:14.098191 15678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.098228 15678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:14.098256 15678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.103899 15678 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41219
I20260926 07:04:14.103969 15806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41219 every 8 connection(s)
I20260926 07:04:14.104281 15678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:14.112283 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15678
I20260926 07:04:14.112424 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:04:14.112648 1893 external_mini_cluster.cc:1428] Setting key f49ed820d987bf31347671c5d7c76567
I20260926 07:04:14.113415 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.105023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:46709 (local address 127.1.217.126:43825)
0926 07:04:14.105065 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.105067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.105274 (+ 207us) server_negotiation.cc:407] Connection header received
0926 07:04:14.105713 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.105714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.105728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.105779 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.106649 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.107171 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.108140 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.108341 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.108397 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.110590 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.110591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.110591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.110598 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.111823 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.112192 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.112194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.112194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.112265 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.112576 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.112577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.112577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.112736 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.112867 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.113198 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.113312 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:04:14.113595 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.104989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:46709)
0926 07:04:14.105153 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.105187 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.105633 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.105829 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.105832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.106387 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.106574 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.106578 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.107247 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.107248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.108023 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.108025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.108416 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.108417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.108521 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.109150 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.109158 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.110502 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.111894 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.111900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.111902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.112111 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.112337 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.112338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.112338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.112522 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.112865 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.112873 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.113055 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.113165 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.113334 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260926 07:04:14.114322 15807 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:14.114442 15807 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:14.114650 15807 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:14.115234 15616 ts_manager.cc:194] Registered new tserver with Master: 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219)
I20260926 07:04:14.116149 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:46709
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:04:14.139248 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:14.222150 15812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.222378 15812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.222417 15812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.223547 15812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.223610 15812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.223654 15812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:14.225087 15812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.225353 15812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.225576 15812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.230523 15818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.230633 15812 server_base.cc:1034] running on GCE node
W20260926 07:04:14.230492 15820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.230492 15817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.230949 15812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.231228 15812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.232407 15812 hybrid_clock.cc:648] HybridClock initialized: now 1790406254232367 us; error 56 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.234830 15812 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:14.235319 15812 webserver.cc:460] Webserver started at http://127.1.217.66:39657/ using document root <none> and password file <none>
I20260926 07:04:14.235554 15812 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.235630 15812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.235776 15812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:14.236881 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "8cda0fbce6346424ca1479ff25908e1f"
server_key_iv: "b2674128254e196d908d00cab8911fa4"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.237263 15812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "8cda0fbce6346424ca1479ff25908e1f"
server_key_iv: "b2674128254e196d908d00cab8911fa4"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.238546 15812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:04:14.239243 15827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.239430 15812 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:14.239530 15812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "8cda0fbce6346424ca1479ff25908e1f"
server_key_iv: "b2674128254e196d908d00cab8911fa4"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.239619 15812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.252815 15812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.254632 15812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.254791 15812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.255012 15812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:14.255317 15812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:14.255369 15812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.255410 15812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:14.255438 15812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.261179 15812 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:38225
I20260926 07:04:14.261226 15940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:38225 every 8 connection(s)
I20260926 07:04:14.261529 15812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:14.263865 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15812
I20260926 07:04:14.263969 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:04:14.264168 1893 external_mini_cluster.cc:1428] Setting key a6f02596cc1e4e0ee03e53d50fbaa435
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:14.271687 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.262296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:60635 (local address 127.1.217.126:43825)
0926 07:04:14.262379 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.262381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.262548 (+ 167us) server_negotiation.cc:407] Connection header received
0926 07:04:14.263014 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.263015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.263025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.263082 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.264173 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.264716 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.265818 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.266029 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.266111 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.268603 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.268605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.268605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.268611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.270114 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.270603 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.270604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.270604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.270655 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.270936 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.270937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.270937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.271082 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.271144 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.271512 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.271612 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 07:04:14.271905 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.262257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:60635)
0926 07:04:14.262433 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.262468 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.262934 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.263142 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.263146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.263765 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.264073 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.264081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.264873 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.264874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.265711 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.265714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.266116 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.266118 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.266258 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.266768 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.266775 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.268498 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.270199 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.270206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.270208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.270523 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.270720 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.270721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.270721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.270864 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.271157 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.271162 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.271339 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.271473 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.271647 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260926 07:04:14.272485 15941 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:14.272590 15941 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
I20260926 07:04:14.272825 15941 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:14.273207 15616 ts_manager.cc:194] Registered new tserver with Master: cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
I20260926 07:04:14.273630 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:60635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:04:14.288950 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:14.368705 15946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.368969 15946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.369005 15946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.370146 15946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.370214 15946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.370260 15946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:14.371676 15946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.371944 15946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.372166 15946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.377172 15954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.377177 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.377177 15951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.377305 15946 server_base.cc:1034] running on GCE node
I20260926 07:04:14.377658 15946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.377907 15946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.379043 15946 hybrid_clock.cc:648] HybridClock initialized: now 1790406254379030 us; error 29 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.381471 15946 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:14.381929 15946 webserver.cc:460] Webserver started at http://127.1.217.67:33847/ using document root <none> and password file <none>
I20260926 07:04:14.382197 15946 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.382279 15946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.382437 15946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:14.383512 15946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "33f2ccc9bb9a4708b89508b757c91a23"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "f09e46eaa4e51b2e837ccc294e06018e"
server_key_iv: "665c03af5cd7be6ad81f29590e5d860f"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.383890 15946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "33f2ccc9bb9a4708b89508b757c91a23"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "f09e46eaa4e51b2e837ccc294e06018e"
server_key_iv: "665c03af5cd7be6ad81f29590e5d860f"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.385421 15946 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:04:14.386154 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.386291 15946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:14.386401 15946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "33f2ccc9bb9a4708b89508b757c91a23"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "f09e46eaa4e51b2e837ccc294e06018e"
server_key_iv: "665c03af5cd7be6ad81f29590e5d860f"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.386483 15946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.398329 15946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.400121 15946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.400288 15946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.400530 15946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:14.400924 15946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:14.400979 15946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.401026 15946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:14.401053 15946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.406989 15946 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:32863
I20260926 07:04:14.407038 16074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:32863 every 8 connection(s)
I20260926 07:04:14.407362 15946 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:04:14.413246 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 15946
I20260926 07:04:14.413317 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:04:14.413494 1893 external_mini_cluster.cc:1428] Setting key dab46cc08ecf3104a956e603642c2ba4
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:14.416594 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.408073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:35129 (local address 127.1.217.126:43825)
0926 07:04:14.408141 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.408143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.408338 (+ 195us) server_negotiation.cc:407] Connection header received
0926 07:04:14.408946 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.408947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.408960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.409003 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.409949 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.410466 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.411461 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.411671 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.411759 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.414093 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.414094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.414095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.414101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.415525 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.415877 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.415878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.415878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.415925 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.416127 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.416128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.416128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.416221 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.416270 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.416475 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.416531 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:04:14.416796 16077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.408056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:35129)
0926 07:04:14.408211 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.408247 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.408841 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.409089 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.409092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.409644 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.409850 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.409855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.410563 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.410564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.411360 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.411362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.411770 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.411770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.411893 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.412448 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.412456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.413984 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.415591 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.415597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.415599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.415813 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.415982 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.415982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.415983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.416075 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.416275 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.416278 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.416385 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.416463 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.416575 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260926 07:04:14.417210 16075 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:14.417302 16075 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:14.417443 16075 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:14.417764 15616 ts_manager.cc:194] Registered new tserver with Master: 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863)
I20260926 07:04:14.418133 15616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:35129
I20260926 07:04:14.425083 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:14.432629 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.426590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.1.217.126:43825)
0926 07:04:14.426649 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.426651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.426768 (+ 117us) server_negotiation.cc:407] Connection header received
0926 07:04:14.426807 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.426808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.426816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.426869 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.427486 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.427954 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.428619 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.428877 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.428931 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.430132 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.430133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.430133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.430140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.431413 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.431827 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.431828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.431828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.431877 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.432086 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.432086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.432087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.432229 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.432314 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.432445 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.432514 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:04:14.434113 15616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:04:14.435612 15616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:04:14.442209 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.1.217.67:32863)
0926 07:04:14.438189 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.438192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.438233 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:04:14.438308 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.438311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.438333 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.438637 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:14.438700 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439767 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441018 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.441765 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441851 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.441916 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.442090 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.442131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":101,"threads_started":1}
I20260926 07:04:14.442230 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:38225 (local address 127.0.0.1:51142)
0926 07:04:14.437849 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.437886 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.437947 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.438771 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.438772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.438773 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:14.438818 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.438820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439663 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.439664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.441024 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.441027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.441879 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.441970 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.441988 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.442069 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.442137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":35,"thread_start_us":55,"threads_started":1}
I20260926 07:04:14.442466 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51142 (local address 127.1.217.66:38225)
0926 07:04:14.438343 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.438348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.438359 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:04:14.438646 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.438649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.438673 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.438775 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:14.438885 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439560 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441111 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.441794 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441872 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.441919 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.442332 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.442381 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":83,"threads_started":1}
I20260926 07:04:14.442703 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.0.0.1:33198)
0926 07:04:14.438179 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.438189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.438240 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.438451 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.438452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.438454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:14.438529 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.438530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439880 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.439880 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.440934 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.440937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.441881 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.441970 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.441988 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.442069 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.442137 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":29,"spinlock_wait_cycles":768,"thread_start_us":69,"threads_started":1}
I20260926 07:04:14.443151 15875 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0686294743471180e399c7cb8444f6 (DEFAULT_TABLE table=test-table [id=a4cf85b15de0423092d3205f39cd1821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:14.443481 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.1.217.65:41219)
0926 07:04:14.438118 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.438122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.438235 (+ 113us) server_negotiation.cc:407] Connection header received
0926 07:04:14.438314 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.438316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.438351 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.438427 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:14.438593 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439742 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.441957 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.443081 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.443158 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.443215 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.443362 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.443408 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260926 07:04:14.443493 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.437701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.0.0.1:53080)
0926 07:04:14.438108 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.438120 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.438231 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.438432 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.438435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.438437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:14.438514 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.438516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.439853 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.439854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.441026 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.441027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.443205 (+ 2178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.443206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.443275 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.443279 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.443325 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.443373 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":10,"thread_start_us":59,"threads_started":1}
I20260926 07:04:14.443575 15875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0686294743471180e399c7cb8444f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:14.444116 15741 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0686294743471180e399c7cb8444f6 (DEFAULT_TABLE table=test-table [id=a4cf85b15de0423092d3205f39cd1821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:14.444415 15741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0686294743471180e399c7cb8444f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:14.445500 16009 tablet_service.cc:1467] Processing CreateTablet for tablet 0d0686294743471180e399c7cb8444f6 (DEFAULT_TABLE table=test-table [id=a4cf85b15de0423092d3205f39cd1821]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:14.445796 16009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d0686294743471180e399c7cb8444f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:14.446774 16089 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Bootstrap starting.
I20260926 07:04:14.447430 16090 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap starting.
I20260926 07:04:14.447546 16089 tablet_bootstrap.cc:654] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:14.448045 16089 log.cc:824] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:14.448057 16090 tablet_bootstrap.cc:654] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:14.448613 16090 log.cc:824] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:14.448979 16093 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Bootstrap starting.
I20260926 07:04:14.449064 16089 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: No bootstrap required, opened a new log
I20260926 07:04:14.449143 16089 ts_tablet_manager.cc:1397] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:14.449555 16093 tablet_bootstrap.cc:654] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:14.450017 16093 log.cc:824] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:14.450652 16093 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: No bootstrap required, opened a new log
I20260926 07:04:14.450732 16093 ts_tablet_manager.cc:1397] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:14.450683 16089 raft_consensus.cc:348] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.450874 16089 raft_consensus.cc:374] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:14.450927 16089 raft_consensus.cc:729] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2a33e80e5a4eb9aa714d6a322f44a4, State: Initialized, Role: FOLLOWER
I20260926 07:04:14.451043 16089 consensus_queue.cc:260] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.451340 16089 ts_tablet_manager.cc:1428] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:14.451371 15941 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
I20260926 07:04:14.451946 16090 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: No bootstrap required, opened a new log
I20260926 07:04:14.452015 16090 ts_tablet_manager.cc:1397] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 07:04:14.452757 16093 raft_consensus.cc:348] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.452966 16093 raft_consensus.cc:374] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:14.453009 16093 raft_consensus.cc:729] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f2ccc9bb9a4708b89508b757c91a23, State: Initialized, Role: FOLLOWER
I20260926 07:04:14.453141 16093 consensus_queue.cc:260] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.453431 16093 ts_tablet_manager.cc:1428] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:04:14.453572 16075 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
I20260926 07:04:14.453981 16090 raft_consensus.cc:348] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.454151 16090 raft_consensus.cc:374] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:14.454201 16090 raft_consensus.cc:729] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9c26daf6a84586a83a1a9e7600e788, State: Initialized, Role: FOLLOWER
I20260926 07:04:14.454334 16090 consensus_queue.cc:260] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.454619 16090 ts_tablet_manager.cc:1428] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 07:04:14.454649 15807 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
I20260926 07:04:14.495121 16095 raft_consensus.cc:482] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:14.495275 16095 raft_consensus.cc:504] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.495656 16095 leader_election.cc:288] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219), 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863)
I20260926 07:04:14.498836 15943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.495820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.1.217.66:52519)
0926 07:04:14.495894 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.495905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.495948 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.496089 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.496090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.496094 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:14.496156 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.496157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.497080 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.497081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.497996 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.497999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.498560 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.498645 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.498653 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.498724 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.498766 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 07:04:14.498836 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.495883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:52519 (local address 127.1.217.65:41219)
0926 07:04:14.495925 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.495926 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.495935 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:14.496007 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.496008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.496016 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.496070 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:14.496241 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.496974 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498099 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.498475 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.498572 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.498732 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.498770 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:04:14.498960 16088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.495886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:60801 (local address 127.1.217.67:32863)
0926 07:04:14.495931 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.495933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.496142 (+ 209us) server_negotiation.cc:407] Connection header received
0926 07:04:14.496190 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.496191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.496201 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.496259 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:14.496412 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.497158 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498239 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.498616 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498699 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.498707 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.498859 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.498895 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:04:14.498975 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.495820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.1.217.66:60801)
0926 07:04:14.496082 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.496090 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.496139 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.496265 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.496266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.496267 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:14.496340 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.496342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.497240 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.497241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.498129 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.498131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.498699 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.498700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.498779 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.498787 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.498852 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.498889 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":97,"thread_start_us":110,"threads_started":1}
I20260926 07:04:14.499390 16029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0686294743471180e399c7cb8444f6" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f2ccc9bb9a4708b89508b757c91a23" is_pre_election: true
I20260926 07:04:14.499442 15761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0686294743471180e399c7cb8444f6" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9c26daf6a84586a83a1a9e7600e788" is_pre_election: true
I20260926 07:04:14.499549 16029 raft_consensus.cc:2454] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 0.
I20260926 07:04:14.499552 15761 raft_consensus.cc:2454] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 0.
I20260926 07:04:14.499759 15830 leader_election.cc:302] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4; no voters:
I20260926 07:04:14.499907 16095 raft_consensus.cc:2781] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:14.499961 16095 raft_consensus.cc:482] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:14.499984 16095 raft_consensus.cc:3037] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:14.500684 16095 raft_consensus.cc:504] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.500840 16095 leader_election.cc:288] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 election: Requested vote from peers 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219), 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863)
I20260926 07:04:14.500984 15761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0686294743471180e399c7cb8444f6" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9c26daf6a84586a83a1a9e7600e788"
I20260926 07:04:14.500989 16029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d0686294743471180e399c7cb8444f6" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f2ccc9bb9a4708b89508b757c91a23"
I20260926 07:04:14.501060 16029 raft_consensus.cc:3037] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:14.501061 15761 raft_consensus.cc:3037] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:14.501953 16029 raft_consensus.cc:2454] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 1.
I20260926 07:04:14.501952 15761 raft_consensus.cc:2454] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 1.
I20260926 07:04:14.502096 15830 leader_election.cc:302] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4; no voters:
I20260926 07:04:14.502182 16095 raft_consensus.cc:2781] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:14.502333 16095 raft_consensus.cc:686] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 LEADER]: Becoming Leader. State: Replica: cc2a33e80e5a4eb9aa714d6a322f44a4, State: Running, Role: LEADER
I20260926 07:04:14.502431 16095 consensus_queue.cc:237] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.503245 15616 catalog_manager.cc:5495] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } health_report { overall_health: UNKNOWN } } }
W20260926 07:04:14.512040 15942 tablet.cc:2369] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:14.513706 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.511548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.1.217.66:38225)
0926 07:04:14.511586 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.511588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.511601 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:04:14.511669 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.511669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.511683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.511742 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:14.511839 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.512314 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.513092 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.513302 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.513362 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.513468 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:14.513555 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:14.513610 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.513629 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.513644 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:04:14.515363 16029 raft_consensus.cc:1261] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Refusing update from remote peer cc2a33e80e5a4eb9aa714d6a322f44a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:14.515393 15761 raft_consensus.cc:1261] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Refusing update from remote peer cc2a33e80e5a4eb9aa714d6a322f44a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:14.515673 16099 consensus_queue.cc:1035] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:14.515787 16095 consensus_queue.cc:1035] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:14.518093 16102 mvcc.cc:205] Tried to move back new op lower bound from 7333504018491588608 to 7333504018442092544. Current Snapshot: MvccSnapshot[applied={T|T < 7333504018491588608}]
I20260926 07:04:14.518639 16104 mvcc.cc:205] Tried to move back new op lower bound from 7333504018491588608 to 7333504018442092544. Current Snapshot: MvccSnapshot[applied={T|T < 7333504018491588608}]
I20260926 07:04:14.519245 16103 mvcc.cc:205] Tried to move back new op lower bound from 7333504018491588608 to 7333504018442092544. Current Snapshot: MvccSnapshot[applied={T|T < 7333504018491588608 or (T in {7333504018491588608})}]
I20260926 07:04:14.520087 15615 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:14.520184 15615 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:14.520943 15615 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Sending DeleteTablet for 3 replicas of tablet 0d0686294743471180e399c7cb8444f6
I20260926 07:04:14.521239 15741 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0686294743471180e399c7cb8444f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:14 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53080
I20260926 07:04:14.521281 15875 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0686294743471180e399c7cb8444f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:14 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:51142
I20260926 07:04:14.521452 16112 tablet_replica.cc:330] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: stopping tablet replica
I20260926 07:04:14.521467 16009 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0686294743471180e399c7cb8444f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:14 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33198
I20260926 07:04:14.521489 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15678
I20260926 07:04:14.521548 16112 raft_consensus.cc:2229] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:14.521651 16112 raft_consensus.cc:2258] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:14.521708 16113 tablet_replica.cc:330] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: stopping tablet replica
I20260926 07:04:14.521798 16113 raft_consensus.cc:2229] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:04:14.521896 16113 raft_consensus.cc:2258] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:14.522330 16113 ts_tablet_manager.cc:1905] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:14.523873 16113 ts_tablet_manager.cc:1918] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:14.523967 16113 log.cc:1196] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/0d0686294743471180e399c7cb8444f6
I20260926 07:04:14.524200 16113 ts_tablet_manager.cc:1939] T 0d0686294743471180e399c7cb8444f6 P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting consensus metadata
I20260926 07:04:14.524547 15603 catalog_manager.cc:4841] TS cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): tablet 0d0686294743471180e399c7cb8444f6 (table test-table [id=a4cf85b15de0423092d3205f39cd1821]) successfully deleted
W20260926 07:04:14.527890 15600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:41219 (error 108)
I20260926 07:04:14.528390 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15812
I20260926 07:04:14.528430 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.528282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.0.0.1:53096)
0926 07:04:14.528354 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.528397 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 07:04:14.529211 15600 catalog_manager.cc:4584] TS 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0d0686294743471180e399c7cb8444f6: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41219: connect: Connection refused (error 111)
I20260926 07:04:14.532917 16114 tablet_replica.cc:330] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: stopping tablet replica
I20260926 07:04:14.533043 16114 raft_consensus.cc:2229] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:14.533193 16114 raft_consensus.cc:2258] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:14.533695 16114 ts_tablet_manager.cc:1905] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:14.534968 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15946
I20260926 07:04:14.539801 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 15584
I20260926 07:04:14.539996 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.539850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.0.0.1:33202)
0926 07:04:14.539918 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.539953 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.67:32863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:04:14.545392 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--webserver_interface=127.1.217.126
--webserver_port=42529
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:14.628345 16116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.628548 16116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.628571 16116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.629806 16116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:14.629853 16116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.629868 16116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.629880 16116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:14.629892 16116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:14.631363 16116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=42529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.631616 16116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.631862 16116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.640563 16125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.640556 16116 server_base.cc:1034] running on GCE node
W20260926 07:04:14.640554 16123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.640554 16122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.641038 16116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.641350 16116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.642489 16116 hybrid_clock.cc:648] HybridClock initialized: now 1790406254642481 us; error 28 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.645068 16116 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:14.645538 16116 webserver.cc:460] Webserver started at http://127.1.217.126:42529/ using document root <none> and password file <none>
I20260926 07:04:14.645777 16116 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.645833 16116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.647222 16116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:14.647866 16132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.648047 16116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:14.648138 16116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d6ff1e6e7d18422c8e2f14472ef2d001"
format_stamp: "Formatted at 2026-09-26 07:04:13 on dist-test-slave-t1mc"
server_key: "36f1eee20370e27ff6e1337b758ab934"
server_key_iv: "bea387760e246348659681f22867705b"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.648525 16116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.662400 16116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.664619 16116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.664769 16116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.668722 16116 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43825
I20260926 07:04:14.668788 16184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43825 every 8 connection(s)
I20260926 07:04:14.669188 16116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:14.670176 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16116
I20260926 07:04:14.670534 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:41219
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=45741
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:14.671320 16185 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:04:14.672258 16185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap starting.
I20260926 07:04:14.680467 16185 log.cc:824] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:14.682673 16185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:14.682987 16185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap complete.
I20260926 07:04:14.685524 16185 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:14.685763 16185 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Initialized, Role: FOLLOWER
I20260926 07:04:14.685940 16185 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:14.686017 16185 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:14.686074 16185 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:14.686124 16185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:04:14.687093 16185 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:14.687208 16185 leader_election.cc:302] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ff1e6e7d18422c8e2f14472ef2d001; no voters:
I20260926 07:04:14.687413 16185 leader_election.cc:288] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:04:14.687446 16189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:04:14.687551 16189 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 LEADER]: Becoming Leader. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Running, Role: LEADER
I20260926 07:04:14.687677 16185 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:14.687700 16189 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:14.688089 16190 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:14.688171 16190 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:14.688380 16191 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ff1e6e7d18422c8e2f14472ef2d001. Latest consensus state: current_term: 2 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:14.688457 16191 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:14.688974 16195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:14.689725 16195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:14.689819 16195 catalog_manager.cc:1259] Loaded cluster ID: 21ed797489914c0fabece88bb27981a6
I20260926 07:04:14.689882 16195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:04:14.690140 16195 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 07:04:14.694557 16195 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 07:04:14.697142 16195 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:14.697337 16195 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Loaded TSK: 0
I20260926 07:04:14.697544 16195 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:04:14.754257 16187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.754459 16187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.754482 16187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.755617 16187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.755656 16187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.755681 16187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:14.757138 16187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:41219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=45741
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.757349 16187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.757586 16187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.762461 16213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.762471 16215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.762472 16212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.762746 16187 server_base.cc:1034] running on GCE node
I20260926 07:04:14.762953 16187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.763209 16187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.764374 16187 hybrid_clock.cc:648] HybridClock initialized: now 1790406254764361 us; error 30 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.766846 16187 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:14.767343 16187 webserver.cc:460] Webserver started at http://127.1.217.65:45741/ using document root <none> and password file <none>
I20260926 07:04:14.767575 16187 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.767630 16187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.769017 16187 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:14.769724 16222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.769906 16187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:14.770032 16187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "7b9c26daf6a84586a83a1a9e7600e788"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "deb4f20af3ad951b1e5c5beffded4f4d"
server_key_iv: "21333fc35bfca45360b50f6fb41575c3"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.770376 16187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.792981 16187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.795254 16187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.795418 16187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.795684 16187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:14.796186 16229 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:04:14.797676 16187 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:04:14.797726 16187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:04:14.797760 16187 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:04:14.798419 16187 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:04:14.798461 16187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:14.798513 16229 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap starting.
I20260926 07:04:14.806138 16187 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41219
I20260926 07:04:14.806242 16336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41219 every 8 connection(s)
I20260926 07:04:14.806782 16187 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:14.810385 16229 log.cc:824] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:14.812793 16229 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:14.813068 16229 tablet_bootstrap.cc:492] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap complete.
I20260926 07:04:14.814293 16229 ts_tablet_manager.cc:1397] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:14.815392 16229 raft_consensus.cc:348] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.815833 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16187
I20260926 07:04:14.815829 16229 raft_consensus.cc:729] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9c26daf6a84586a83a1a9e7600e788, State: Initialized, Role: FOLLOWER
I20260926 07:04:14.815969 16229 consensus_queue.cc:260] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } }
I20260926 07:04:14.816118 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:38225
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=39657
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:14.816298 16229 ts_tablet_manager.cc:1428] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260926 07:04:14.820709 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.807628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:39303 (local address 127.1.217.126:43825)
0926 07:04:14.807832 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.807840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.807876 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:04:14.808792 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.808800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.808859 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.808999 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.809968 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.811326 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.812759 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.813338 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.813439 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.817279 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.817283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.817284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.817299 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.818976 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.819494 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.819496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.819496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.819580 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.819867 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.819867 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.819868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.820065 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.820126 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.820478 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.820575 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":86,"threads_started":1}
I20260926 07:04:14.820902 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.807536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:39303)
0926 07:04:14.807729 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.807772 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.808512 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.809080 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.809084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.809493 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.809829 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.809838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.811444 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.811445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.812651 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.812654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.813443 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.813445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.813590 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.814189 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.814199 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.817133 (+ 2934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.819062 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.819070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.819072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.819402 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.819648 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.819649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.819650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.819796 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.820135 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.820144 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.820313 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.820438 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.820624 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":111,"threads_started":1}
I20260926 07:04:14.821600 16337 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:14.821705 16337 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:14.821877 16337 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:14.822630 16149 ts_manager.cc:194] Registered new tserver with Master: 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219)
I20260926 07:04:14.823923 16149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:39303
I20260926 07:04:14.824880 16337 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:04:14.832314 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.823274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.0.0.1:53106)
0926 07:04:14.824169 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.824220 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.824285 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.824577 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.824584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.824981 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.825131 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.825134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.826112 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.826113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.827086 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.827089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.827539 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.827540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.827615 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.828017 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.828021 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.829451 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.831121 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.831127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.831130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.831333 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.831583 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.831584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.831585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.831653 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.831936 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.831942 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.831993 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.832090 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.832215 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":869,"thread_start_us":85,"threads_started":1}
I20260926 07:04:14.832311 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.823356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.1.217.65:41219)
0926 07:04:14.824420 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.824425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.824434 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:14.824466 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.824467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.824485 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.824552 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.825202 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.826029 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.827176 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.827468 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.827520 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.829549 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.829556 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.829556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.829563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.831002 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.831430 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.831432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.831433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.831491 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.831717 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.831718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.831719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.831856 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.831943 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.832126 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.832214 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1036,"thread_start_us":67,"threads_started":1}
I20260926 07:04:14.832914 16271 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d0686294743471180e399c7cb8444f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:14 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53106
I20260926 07:04:14.833432 16346 tablet_replica.cc:330] stopping tablet replica
I20260926 07:04:14.833554 16346 raft_consensus.cc:2229] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:14.833642 16346 raft_consensus.cc:2258] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:14.833931 16346 ts_tablet_manager.cc:1905] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:14.835847 16346 ts_tablet_manager.cc:1918] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:14.835932 16346 log.cc:1196] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/0d0686294743471180e399c7cb8444f6
I20260926 07:04:14.836138 16346 ts_tablet_manager.cc:1939] T 0d0686294743471180e399c7cb8444f6 P 7b9c26daf6a84586a83a1a9e7600e788: Deleting consensus metadata
I20260926 07:04:14.836958 16133 catalog_manager.cc:4841] TS 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219): tablet 0d0686294743471180e399c7cb8444f6 (table test-table [id=a4cf85b15de0423092d3205f39cd1821]) successfully deleted
W20260926 07:04:14.899293 16343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:14.899534 16343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:14.899573 16343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:14.900698 16343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:14.900766 16343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:14.900846 16343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:14.902275 16343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:38225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=39657
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:14.902549 16343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:14.902776 16343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:14.907722 16351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:14.907765 16352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.907918 16343 server_base.cc:1034] running on GCE node
W20260926 07:04:14.907769 16354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:14.908209 16343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:14.908532 16343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:14.909694 16343 hybrid_clock.cc:648] HybridClock initialized: now 1790406254909674 us; error 53 us; skew 500 ppm
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:14.912242 16343 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:14.912757 16343 webserver.cc:460] Webserver started at http://127.1.217.66:39657/ using document root <none> and password file <none>
I20260926 07:04:14.913064 16343 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:14.913147 16343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:14.914525 16343 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:04:14.915086 16361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.915256 16343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:14.915397 16343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "8cda0fbce6346424ca1479ff25908e1f"
server_key_iv: "b2674128254e196d908d00cab8911fa4"
server_key_version: "encryptionkey@0"
I20260926 07:04:14.915798 16343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:14.937026 16343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:14.939450 16343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:14.939620 16343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:14.939910 16343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:14.940302 16343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:14.940423 16343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.940464 16343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:14.940487 16343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:14.946678 16343 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:38225
I20260926 07:04:14.946728 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:38225 every 8 connection(s)
I20260926 07:04:14.947063 16343 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:14.950860 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16343
I20260926 07:04:14.951150 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:32863
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=33847
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:04:14 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406254, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:14.957816 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.947763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:50431)
0926 07:04:14.947930 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:14.947965 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:14.948509 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:14.948796 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:14.948805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:14.949387 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:14.949588 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.949593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.950326 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.950327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.951427 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:14.951430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.951867 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.951868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:14.952033 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.952617 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:14.952626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:14.954345 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:14.956366 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.956372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.956374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.956613 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.956856 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:14.956857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:14.956857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:14.956969 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:14.957285 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:14.957291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:14.957434 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:04:14.957524 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:04:14.957651 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":76,"threads_started":1}
I20260926 07:04:14.957820 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:14.947761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50431 (local address 127.1.217.126:43825)
0926 07:04:14.947834 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:14.947836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:14.948054 (+ 218us) server_negotiation.cc:407] Connection header received
0926 07:04:14.948625 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:14.948627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:14.948643 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:14.948703 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:14.949699 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.950214 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.951518 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:14.951748 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:14.951859 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:14.954445 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:14.954446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:14.954447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:14.954456 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:14.956269 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.956690 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.956691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.956692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.956760 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:14.957043 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:14.957044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:14.957044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:14.957203 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:14.957287 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:14.957573 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:04:14.957681 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:04:14.958401 16475 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:14.958518 16475 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:14.958734 16475 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:14.959172 16149 ts_manager.cc:194] Registered new tserver with Master: cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
I20260926 07:04:14.959601 16149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:50431
W20260926 07:04:15.034708 16478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:15.034899 16478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:15.034922 16478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:15.036059 16478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:15.036108 16478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:15.036170 16478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:15.037774 16478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:32863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=33847
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:15.038002 16478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:15.038219 16478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:15.043150 16484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.043217 16478 server_base.cc:1034] running on GCE node
W20260926 07:04:15.043143 16486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.043200 16483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.043524 16478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:15.043718 16478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:15.044879 16478 hybrid_clock.cc:648] HybridClock initialized: now 1790406255044858 us; error 44 us; skew 500 ppm
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:15.047319 16478 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:15.047773 16478 webserver.cc:460] Webserver started at http://127.1.217.67:33847/ using document root <none> and password file <none>
I20260926 07:04:15.048017 16478 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:15.048100 16478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:15.049435 16478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:15.050079 16493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.050248 16478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:15.050341 16478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "33f2ccc9bb9a4708b89508b757c91a23"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "f09e46eaa4e51b2e837ccc294e06018e"
server_key_iv: "665c03af5cd7be6ad81f29590e5d860f"
server_key_version: "encryptionkey@0"
I20260926 07:04:15.050681 16478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:15.082841 16478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:15.084842 16478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:15.085021 16478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:15.085269 16478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:15.085701 16500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 07:04:15.087154 16500 ts_tablet_manager.cc:1852] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 07:04:15.087411 16500 ts_tablet_manager.cc:1905] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:15.089857 16500 ts_tablet_manager.cc:1918] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:15.089949 16500 log.cc:1196] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/0d0686294743471180e399c7cb8444f6
I20260926 07:04:15.090215 16500 ts_tablet_manager.cc:1939] T 0d0686294743471180e399c7cb8444f6 P 33f2ccc9bb9a4708b89508b757c91a23: Deleting consensus metadata
I20260926 07:04:15.090423 16478 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 07:04:15.090487 16478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 07:04:15.090535 16478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:15.090567 16478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.096308 16478 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:32863
I20260926 07:04:15.096364 16607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:32863 every 8 connection(s)
I20260926 07:04:15.096654 16478 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:15.105018 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.097380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:46137 (local address 127.1.217.126:43825)
0926 07:04:15.097424 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.097426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.097649 (+ 223us) server_negotiation.cc:407] Connection header received
0926 07:04:15.098103 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.098105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.098115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.098167 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:15.098721 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.099199 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.100122 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.100297 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.100345 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.102532 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:15.102533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:15.102534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:15.102540 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:15.103752 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:15.104116 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:15.104117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:15.104117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:15.104166 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:15.104438 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:15.104439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:15.104439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:15.104539 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:15.104588 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.104877 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.104947 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:15.105206 16610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.097373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:46137)
0926 07:04:15.097529 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.097564 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.098024 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.098219 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:15.098222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:15.098466 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:15.098649 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.098653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.099272 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.099273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.100053 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.100055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.100365 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.100365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.100461 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.101025 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:15.101034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:15.102434 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:15.103830 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:15.103836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:15.103837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:15.104050 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:15.104229 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:15.104230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:15.104230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:15.104361 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:15.104596 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:15.104601 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:15.104716 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 07:04:15.104859 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 07:04:15.104986 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 07:04:15.105829 16608 heartbeater.cc:347] Connected to a master server at 127.1.217.126:43825
I20260926 07:04:15.105939 16608 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:15.106118 16608 heartbeater.cc:511] Master 127.1.217.126:43825 requested a full tablet report, sending...
I20260926 07:04:15.106496 16149 ts_manager.cc:194] Registered new tserver with Master: 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863)
I20260926 07:04:15.106724 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16478
I20260926 07:04:15.106984 16149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:46137
I20260926 07:04:15.112840 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.107064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60434 (local address 127.1.217.126:43825)
0926 07:04:15.107141 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.107143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.107281 (+ 138us) server_negotiation.cc:407] Connection header received
0926 07:04:15.107357 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.107358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.107368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.107423 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:15.107969 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.108421 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.109227 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.109412 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.109505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.110614 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:15.110615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:15.110615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:15.110622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:15.111702 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:15.112054 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:15.112055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:15.112055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:15.112112 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:15.112312 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:15.112313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:15.112313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:15.112424 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:15.112610 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.112646 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.112716 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 07:04:15.113631 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:15.116510 16340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.113953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.1.217.126:43825)
0926 07:04:15.114004 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.114006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.114123 (+ 117us) server_negotiation.cc:407] Connection header received
0926 07:04:15.114198 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.114198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.114235 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.114320 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:15.114426 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.114893 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.115642 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.115842 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.115924 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.116115 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:15.116244 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:15.116290 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.116421 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.116439 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 07:04:15.116832 16149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:04:15.118420 16149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:04:15.120631 16271 tablet_service.cc:1467] Processing CreateTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c (DEFAULT_TABLE table=test-table [id=a58b8230825744b0bd4ea12925adb423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:15.120874 16271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8265dadaac6e4edabf0f3fc0c5727d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:15.122462 16229 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap starting.
I20260926 07:04:15.122797 16229 tablet_bootstrap.cc:654] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:15.123289 16229 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: No bootstrap required, opened a new log
I20260926 07:04:15.123355 16229 ts_tablet_manager.cc:1397] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:15.123689 16229 raft_consensus.cc:348] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.123777 16229 raft_consensus.cc:374] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:15.123831 16229 raft_consensus.cc:729] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9c26daf6a84586a83a1a9e7600e788, State: Initialized, Role: FOLLOWER
I20260926 07:04:15.123875 16229 consensus_queue.cc:260] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.123981 16229 ts_tablet_manager.cc:1428] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:15.124287 16613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.120391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:38225 (local address 127.0.0.1:51156)
0926 07:04:15.120491 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.120502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.120564 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.121133 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:15.121134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:15.121136 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:15.121185 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.121187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.122089 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.122091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.123117 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.123119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.124048 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.124049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.124111 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.124135 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 07:04:15.124168 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 07:04:15.124212 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260926 07:04:15.124692 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.120814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.1.217.66:38225)
0926 07:04:15.120955 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.120959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.120969 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:04:15.121010 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.121011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.121025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.121114 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:15.121241 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.121989 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.123213 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.123943 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.124466 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.124539 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.124567 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.124614 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260926 07:04:15.125324 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.120593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.1.217.67:32863)
0926 07:04:15.120865 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.120871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.120882 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:04:15.121021 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.121023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.121041 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.121260 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:15.121315 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.122334 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.124160 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.125010 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.125081 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.125129 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.125238 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.125266 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":104,"threads_started":1}
I20260926 07:04:15.125335 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.120381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.0.0.1:33214)
0926 07:04:15.120430 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.120442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.120478 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.121149 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:15.121150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:15.121152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:15.121194 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.121196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.123013 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.123014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.124080 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.124082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.125093 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.125094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.125162 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.125166 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:04:15.125230 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:04:15.125275 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:04:15.125408 16409 tablet_service.cc:1467] Processing CreateTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c (DEFAULT_TABLE table=test-table [id=a58b8230825744b0bd4ea12925adb423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:15.125762 16409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8265dadaac6e4edabf0f3fc0c5727d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:15.125862 16542 tablet_service.cc:1467] Processing CreateTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c (DEFAULT_TABLE table=test-table [id=a58b8230825744b0bd4ea12925adb423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:15.126161 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8265dadaac6e4edabf0f3fc0c5727d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:15.127857 16500 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Bootstrap starting.
I20260926 07:04:15.128679 16500 tablet_bootstrap.cc:654] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:15.128960 16617 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Bootstrap starting.
I20260926 07:04:15.129153 16500 log.cc:824] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:15.129544 16617 tablet_bootstrap.cc:654] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:15.129840 16500 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: No bootstrap required, opened a new log
I20260926 07:04:15.129899 16500 ts_tablet_manager.cc:1397] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:15.130164 16617 log.cc:824] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:15.130851 16617 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: No bootstrap required, opened a new log
I20260926 07:04:15.130913 16617 ts_tablet_manager.cc:1397] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:15.131532 16500 raft_consensus.cc:348] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.131685 16500 raft_consensus.cc:374] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:15.131729 16500 raft_consensus.cc:729] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f2ccc9bb9a4708b89508b757c91a23, State: Initialized, Role: FOLLOWER
I20260926 07:04:15.131857 16500 consensus_queue.cc:260] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.132124 16500 ts_tablet_manager.cc:1428] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:15.132156 16608 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
I20260926 07:04:15.132575 16617 raft_consensus.cc:348] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.132750 16617 raft_consensus.cc:374] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:15.132826 16617 raft_consensus.cc:729] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2a33e80e5a4eb9aa714d6a322f44a4, State: Initialized, Role: FOLLOWER
I20260926 07:04:15.132946 16617 consensus_queue.cc:260] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.133236 16475 heartbeater.cc:503] Master 127.1.217.126:43825 was elected leader, sending a full tablet report...
I20260926 07:04:15.133498 16617 ts_tablet_manager.cc:1428] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260926 07:04:15.197593 16476 tablet.cc:2369] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:15.216456 16621 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:15.216572 16621 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.217027 16621 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219)
I20260926 07:04:15.220086 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.217285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:49975 (local address 127.1.217.67:32863)
0926 07:04:15.217385 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.217386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.217392 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:04:15.217415 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.217416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.217425 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.217495 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:15.217679 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.218163 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219291 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.219734 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219818 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.219827 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.219984 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.220019 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 07:04:15.220077 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.217218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.1.217.66:49975)
0926 07:04:15.217258 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.217270 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.217311 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.217514 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:15.217515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:15.217516 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:15.217598 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.217600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.218248 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.218249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.219204 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.219207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.219815 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.219899 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.219907 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:04:15.219967 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:04:15.220004 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":19}
I20260926 07:04:15.220150 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.217287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:56337 (local address 127.1.217.65:41219)
0926 07:04:15.217347 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.217351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.217500 (+ 149us) server_negotiation.cc:407] Connection header received
0926 07:04:15.217562 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.217564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.217593 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.217665 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:15.217816 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.218306 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219301 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.219816 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219907 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.219986 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.220035 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.220075 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:04:15.220204 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.217218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.1.217.66:56337)
0926 07:04:15.217404 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.217412 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.217505 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.217666 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:15.217667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:15.217669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:15.217738 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.217740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.218407 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.218408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.219207 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:15.219209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.219908 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.219909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:15.219974 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.219979 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:04:15.220030 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 07:04:15.220067 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":60,"thread_start_us":93,"threads_started":1}
I20260926 07:04:15.220551 16291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8265dadaac6e4edabf0f3fc0c5727d0c" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9c26daf6a84586a83a1a9e7600e788" is_pre_election: true
I20260926 07:04:15.220707 16291 raft_consensus.cc:2454] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 0.
I20260926 07:04:15.220649 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8265dadaac6e4edabf0f3fc0c5727d0c" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f2ccc9bb9a4708b89508b757c91a23" is_pre_election: true
I20260926 07:04:15.220877 16562 raft_consensus.cc:2454] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 0.
I20260926 07:04:15.220979 16362 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788, cc2a33e80e5a4eb9aa714d6a322f44a4; no voters:
I20260926 07:04:15.221117 16621 raft_consensus.cc:2781] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:15.221169 16621 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:15.221192 16621 raft_consensus.cc:3037] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:15.221837 16621 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.221975 16621 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 election: Requested vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219)
I20260926 07:04:15.222103 16562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8265dadaac6e4edabf0f3fc0c5727d0c" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f2ccc9bb9a4708b89508b757c91a23"
I20260926 07:04:15.222116 16291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8265dadaac6e4edabf0f3fc0c5727d0c" candidate_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b9c26daf6a84586a83a1a9e7600e788"
I20260926 07:04:15.222180 16562 raft_consensus.cc:3037] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:15.222208 16291 raft_consensus.cc:3037] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:15.222823 16562 raft_consensus.cc:2454] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 1.
I20260926 07:04:15.222983 16291 raft_consensus.cc:2454] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc2a33e80e5a4eb9aa714d6a322f44a4 in term 1.
I20260926 07:04:15.222995 16364 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4; no voters:
I20260926 07:04:15.223104 16621 raft_consensus.cc:2781] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:15.223258 16621 raft_consensus.cc:686] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 LEADER]: Becoming Leader. State: Replica: cc2a33e80e5a4eb9aa714d6a322f44a4, State: Running, Role: LEADER
I20260926 07:04:15.223373 16621 consensus_queue.cc:237] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.224169 16149 catalog_manager.cc:5495] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } health_report { overall_health: UNKNOWN } } }
I20260926 07:04:15.243013 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.240916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.1.217.66:38225)
0926 07:04:15.240975 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:15.240977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:15.240984 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:04:15.241016 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:15.241016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:15.241033 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:15.241097 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:15.241228 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.241694 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.242360 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:15.242539 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:15.242592 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:15.242731 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:15.242823 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:15.242879 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:15.242915 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 07:04:15.242931 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:04:15.244812 16562 raft_consensus.cc:1261] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Refusing update from remote peer cc2a33e80e5a4eb9aa714d6a322f44a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:15.244835 16291 raft_consensus.cc:1261] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Refusing update from remote peer cc2a33e80e5a4eb9aa714d6a322f44a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:15.245198 16621 consensus_queue.cc:1035] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:15.245332 16623 consensus_queue.cc:1035] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:15.247228 16625 mvcc.cc:205] Tried to move back new op lower bound from 7333504021479387136 to 7333504021395132416. Current Snapshot: MvccSnapshot[applied={T|T < 7333504021479387136}]
I20260926 07:04:15.247485 16626 mvcc.cc:205] Tried to move back new op lower bound from 7333504021479387136 to 7333504021395132416. Current Snapshot: MvccSnapshot[applied={T|T < 7333504021479387136}]
I20260926 07:04:15.249857 16149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:15.249958 16149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:15.250667 16149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Sending DeleteTablet for 3 replicas of tablet 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:15.251029 16271 tablet_service.cc:1514] Processing DeleteTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:15 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53106
I20260926 07:04:15.251012 16542 tablet_service.cc:1514] Processing DeleteTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:15 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:33214
I20260926 07:04:15.251044 16409 tablet_service.cc:1514] Processing DeleteTablet for tablet 8265dadaac6e4edabf0f3fc0c5727d0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:15 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:51156
I20260926 07:04:15.251121 16346 tablet_replica.cc:330] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: stopping tablet replica
I20260926 07:04:15.251164 16346 raft_consensus.cc:2229] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:15.251214 16346 raft_consensus.cc:2258] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:15.251289 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16187
I20260926 07:04:15.251421 16635 tablet_replica.cc:330] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: stopping tablet replica
I20260926 07:04:15.251497 16346 ts_tablet_manager.cc:1905] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:15.251601 16635 raft_consensus.cc:2229] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:04:15.251729 16635 raft_consensus.cc:2258] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:15.252193 16635 ts_tablet_manager.cc:1905] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:15.254212 16635 ts_tablet_manager.cc:1918] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:15.254283 16635 log.cc:1196] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:15.254601 16635 ts_tablet_manager.cc:1939] T 8265dadaac6e4edabf0f3fc0c5727d0c P cc2a33e80e5a4eb9aa714d6a322f44a4: Deleting consensus metadata
I20260926 07:04:15.255329 16136 catalog_manager.cc:4841] TS cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): tablet 8265dadaac6e4edabf0f3fc0c5727d0c (table test-table [id=a58b8230825744b0bd4ea12925adb423]) successfully deleted
I20260926 07:04:15.256915 16636 tablet_replica.cc:330] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: stopping tablet replica
I20260926 07:04:15.257027 16636 raft_consensus.cc:2229] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:15.257135 16636 raft_consensus.cc:2258] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:15.257561 16636 ts_tablet_manager.cc:1905] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:04:15.258281 16133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:41219 (error 108)
I20260926 07:04:15.258481 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16343
I20260926 07:04:15.259153 16636 ts_tablet_manager.cc:1918] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:15.259200 16636 log.cc:1196] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:15.259454 16636 ts_tablet_manager.cc:1939] T 8265dadaac6e4edabf0f3fc0c5727d0c P 33f2ccc9bb9a4708b89508b757c91a23: Deleting consensus metadata
I20260926 07:04:15.259832 16135 catalog_manager.cc:4841] TS 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): tablet 8265dadaac6e4edabf0f3fc0c5727d0c (table test-table [id=a58b8230825744b0bd4ea12925adb423]) successfully deleted
I20260926 07:04:15.261660 16344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.261521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41219 (local address 127.0.0.1:53122)
0926 07:04:15.261582 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.261626 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 07:04:15.261775 16133 catalog_manager.cc:4584] TS 7b9c26daf6a84586a83a1a9e7600e788 (127.1.217.65:41219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8265dadaac6e4edabf0f3fc0c5727d0c: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41219: connect: Connection refused (error 111)
I20260926 07:04:15.263300 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16478
I20260926 07:04:15.268286 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16116
I20260926 07:04:15.273399 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--webserver_interface=127.1.217.126
--webserver_port=42529
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:15.354238 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:15.354431 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:15.354452 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:15.355515 16638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:15.355551 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:15.355564 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:15.355576 16638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:15.355588 16638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:15.356952 16638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=42529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:15.357182 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:15.357374 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:15.362298 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.362294 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.362294 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.362313 16638 server_base.cc:1034] running on GCE node
I20260926 07:04:15.362771 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:15.363036 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:15.364182 16638 hybrid_clock.cc:648] HybridClock initialized: now 1790406255364171 us; error 26 us; skew 500 ppm
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:15.366645 16638 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:15.367122 16638 webserver.cc:460] Webserver started at http://127.1.217.126:42529/ using document root <none> and password file <none>
I20260926 07:04:15.367362 16638 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:15.367408 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:15.368723 16638 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:04:15.369403 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.369591 16638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:15.369688 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d6ff1e6e7d18422c8e2f14472ef2d001"
format_stamp: "Formatted at 2026-09-26 07:04:13 on dist-test-slave-t1mc"
server_key: "36f1eee20370e27ff6e1337b758ab934"
server_key_iv: "bea387760e246348659681f22867705b"
server_key_version: "encryptionkey@0"
I20260926 07:04:15.370016 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:15.387867 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:15.389804 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:15.389910 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:15.393641 16638 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43825
I20260926 07:04:15.393698 16705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43825 every 8 connection(s)
I20260926 07:04:15.394068 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:15.395620 16706 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:04:15.396195 16706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap starting.
I20260926 07:04:15.397832 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16638
I20260926 07:04:15.398177 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:41219
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=45741
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:15.404846 16706 log.cc:824] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:15.407071 16706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:15.407392 16706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001: Bootstrap complete.
I20260926 07:04:15.409718 16706 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:15.409940 16706 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Initialized, Role: FOLLOWER
I20260926 07:04:15.410109 16706 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:15.410166 16706 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:15.410228 16706 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:15.410282 16706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:04:15.411258 16706 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:15.411365 16706 leader_election.cc:302] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ff1e6e7d18422c8e2f14472ef2d001; no voters:
I20260926 07:04:15.411561 16706 leader_election.cc:288] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:04:15.411615 16710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:04:15.411727 16710 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [term 3 LEADER]: Becoming Leader. State: Replica: d6ff1e6e7d18422c8e2f14472ef2d001, State: Running, Role: LEADER
I20260926 07:04:15.411829 16706 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:15.411823 16710 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } }
I20260926 07:04:15.412215 16711 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ff1e6e7d18422c8e2f14472ef2d001. Latest consensus state: current_term: 3 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:15.412288 16711 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:15.412521 16714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:15.413086 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ff1e6e7d18422c8e2f14472ef2d001" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43825 } } }
I20260926 07:04:15.413221 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ff1e6e7d18422c8e2f14472ef2d001 [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:15.413342 16714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:15.413460 16714 catalog_manager.cc:1259] Loaded cluster ID: 21ed797489914c0fabece88bb27981a6
I20260926 07:04:15.413496 16714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 07:04:15.413703 16714 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f69b88209ad google::LogMessage::Fail() at ??:0
@ 0x7f69b882388a google::LogMessage::SendToLog() at ??:0
@ 0x7f69b88204de google::LogMessage::Flush() at ??:0
@ 0x7f69b8821a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f69c07410f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f69c074a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f69ba6ee0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f69ba6e3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f69bb5476db start_thread at ??:0
@ 0x7f69b731e61f clone at ??:0
W20260926 07:04:15.491020 16708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:15.491210 16708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:15.491232 16708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:15.492368 16708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:15.492408 16708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:15.492434 16708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:15.493911 16708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:41219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=45741
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:15.494150 16708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:15.494382 16708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:15.499482 16745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.499490 16742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.499495 16743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.499665 16708 server_base.cc:1034] running on GCE node
I20260926 07:04:15.499910 16708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:15.500169 16708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:15.501358 16708 hybrid_clock.cc:648] HybridClock initialized: now 1790406255501337 us; error 50 us; skew 500 ppm
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:15.504434 16708 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:15.504982 16708 webserver.cc:460] Webserver started at http://127.1.217.65:45741/ using document root <none> and password file <none>
I20260926 07:04:15.505261 16708 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:15.505323 16708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:15.507068 16708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:04:15.507923 16752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.508095 16708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:15.508181 16708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "7b9c26daf6a84586a83a1a9e7600e788"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "deb4f20af3ad951b1e5c5beffded4f4d"
server_key_iv: "21333fc35bfca45360b50f6fb41575c3"
server_key_version: "encryptionkey@0"
I20260926 07:04:15.508496 16708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:15.525672 16708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:15.527973 16708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:15.528108 16708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:15.528366 16708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:15.528843 16759 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:04:15.530875 16708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:04:15.530937 16708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:04:15.530979 16708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:04:15.531584 16708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:04:15.531648 16708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:15.531718 16759 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap starting.
I20260926 07:04:15.533627 16759 log.cc:824] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:15.536489 16759 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:15.536818 16759 tablet_bootstrap.cc:492] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Bootstrap complete.
I20260926 07:04:15.536964 16759 ts_tablet_manager.cc:1397] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 07:04:15.538792 16708 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41219
I20260926 07:04:15.538842 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41219 every 8 connection(s)
I20260926 07:04:15.538816 16759 raft_consensus.cc:348] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.539129 16708 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:15.539160 16759 raft_consensus.cc:729] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9c26daf6a84586a83a1a9e7600e788, State: Initialized, Role: FOLLOWER
I20260926 07:04:15.539355 16759 consensus_queue.cc:260] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:15.539755 16759 ts_tablet_manager.cc:1428] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:04:15.543524 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16708
I20260926 07:04:15.543812 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:38225
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=39657
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:15.615072 16871 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:04:15.539741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:44841)
0926 07:04:15.539918 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.539960 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:15.540092 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:15.614865 (+ 74773us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
W20260926 07:04:15.615286 16868 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:43825 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:43825: Client connection negotiation failed: client connection to 127.1.217.126:43825: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:04:15.615623 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.615503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:44239)
0926 07:04:15.615561 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.615604 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:04:15.615955 16871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.615876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:50939)
0926 07:04:15.615920 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.615941 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:04:15.616076 16868 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:04:15.628958 16872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:15.629153 16872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:15.629175 16872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:15.630318 16872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:15.630358 16872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:15.630412 16872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:15.631832 16872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:38225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=39657
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:15.632064 16872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:15.632282 16872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:15.637037 16878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.637079 16880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.637038 16877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.637391 16872 server_base.cc:1034] running on GCE node
I20260926 07:04:15.637607 16872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:15.637837 16872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:15.638998 16872 hybrid_clock.cc:648] HybridClock initialized: now 1790406255638983 us; error 28 us; skew 500 ppm
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:15.641412 16872 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:15.641840 16872 webserver.cc:460] Webserver started at http://127.1.217.66:39657/ using document root <none> and password file <none>
I20260926 07:04:15.642059 16872 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:15.642103 16872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:15.643399 16872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:15.643988 16887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.644151 16872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.644236 16872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "8cda0fbce6346424ca1479ff25908e1f"
server_key_iv: "b2674128254e196d908d00cab8911fa4"
server_key_version: "encryptionkey@0"
I20260926 07:04:15.644548 16872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:15.668082 16872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:15.670501 16872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:15.670619 16872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:15.670814 16872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:15.671149 16872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:15.671183 16872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.671206 16872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:15.671257 16872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.677042 16872 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:38225
I20260926 07:04:15.677127 17000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:38225 every 8 connection(s)
I20260926 07:04:15.677474 16872 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:15.678524 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.678164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:44155)
0926 07:04:15.678319 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.678387 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260926 07:04:15.678923 17001 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:43825 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:43825: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
I20260926 07:04:15.679181 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.679089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:40931)
0926 07:04:15.679139 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.679164 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:04:15.679482 17003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.679407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:48379)
0926 07:04:15.679448 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.679469 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 07:04:15.679494 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 16872
W20260926 07:04:15.679607 17001 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:04:15.679772 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:32863
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:43825
--webserver_port=33847
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:15.758608 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:15.758795 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:15.758818 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:15.759922 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:15.759961 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:15.759987 17004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:15.761482 17004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:32863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=33847
--tserver_master_addrs=127.1.217.126:43825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:15.761700 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:15.761942 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:15.766913 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.766923 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:15.766906 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:15.767143 17004 server_base.cc:1034] running on GCE node
I20260926 07:04:15.767364 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:15.767603 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:15.768744 17004 hybrid_clock.cc:648] HybridClock initialized: now 1790406255768734 us; error 30 us; skew 500 ppm
Sep 26 07:04:15 dist-test-slave-t1mc krb5kdc[15571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:15.771070 17004 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:15.771526 17004 webserver.cc:460] Webserver started at http://127.1.217.67:33847/ using document root <none> and password file <none>
I20260926 07:04:15.771768 17004 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:15.771836 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:15.773198 17004 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:15.773928 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.774111 17004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:15.774209 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "33f2ccc9bb9a4708b89508b757c91a23"
format_stamp: "Formatted at 2026-09-26 07:04:14 on dist-test-slave-t1mc"
server_key: "f09e46eaa4e51b2e837ccc294e06018e"
server_key_iv: "665c03af5cd7be6ad81f29590e5d860f"
server_key_version: "encryptionkey@0"
I20260926 07:04:15.774565 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:15.789474 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:15.791826 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:15.791994 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:15.792234 17004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:15.792536 17004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:15.792587 17004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.792629 17004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:15.792655 17004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:15.798450 17004 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:32863
I20260926 07:04:15.798501 17132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:32863 every 8 connection(s)
I20260926 07:04:15.798825 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:04:15.799855 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.799486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:36415)
0926 07:04:15.799646 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.799718 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
W20260926 07:04:15.800289 17133 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:43825 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:43825: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
I20260926 07:04:15.800586 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.800497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:57631)
0926 07:04:15.800542 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.800569 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 07:04:15.800891 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:15.800830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:54125)
0926 07:04:15.800860 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:15.800879 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 07:04:15.800997 17133 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:04:15.804386 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17004
I20260926 07:04:16.616657 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:16.616414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:50777)
0926 07:04:16.616567 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:16.616603 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260926 07:04:16.680199 17138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:16.679891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:47809)
0926 07:04:16.680113 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:16.680147 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":80,"threads_started":1}
I20260926 07:04:16.801596 17139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:16.801397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:60129)
0926 07:04:16.801511 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:16.801544 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 07:04:17.033262 17140 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:17.033432 17140 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:17.033716 17140 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
Sep 26 07:04:17 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.67@KRBTEST.COM
Sep 26 07:04:17 dist-test-slave-t1mc krb5kdc[15571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406255, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.66@KRBTEST.COM
I20260926 07:04:17.044806 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.033914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:38225 (local address 127.1.217.65:56711)
0926 07:04:17.034062 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:17.034072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:17.034110 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:17.035141 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:17.035144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:17.035628 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:17.035977 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:17.035984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.037205 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.037206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:17.038074 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:17.038076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.038671 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:17.038806 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:17.039398 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:17.039406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:17.041407 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:17.043248 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:17.043249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:17.043492 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:17.043748 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:17.043750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:17.043832 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:17.044225 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:17.044231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:17.044367 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 07:04:17.044474 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:04:17.044671 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":126,"thread_start_us":58,"threads_started":1}
I20260926 07:04:17.044806 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.034039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:37833 (local address 127.1.217.67:32863)
0926 07:04:17.034202 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:17.034214 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:17.034250 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:04:17.034835 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:17.034839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:17.034902 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:17.035074 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:17.036159 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.037125 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038169 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.038577 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038692 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:17.041121 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:17.041125 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:17.041126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:17.041137 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:17.043031 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043516 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:17.043520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:17.043520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:17.043632 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043884 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:17.043886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:17.043886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:17.044149 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:17.044244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:17.044527 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 07:04:17.044657 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260926 07:04:17.044911 17143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.034048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:56711 (local address 127.1.217.66:38225)
0926 07:04:17.034201 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:17.034214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:17.034250 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:04:17.034836 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:17.034839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:17.034902 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:17.035074 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:17.036110 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.037104 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038177 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.038575 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038696 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:17.041513 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:17.041517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:17.041517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:17.041529 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:17.043149 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043570 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:17.043573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:17.043573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:17.043653 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043914 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:17.043915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:17.043916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:17.044149 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:17.044244 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:17.044546 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 07:04:17.044695 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":52,"threads_started":1}
I20260926 07:04:17.044978 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.033914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:32863 (local address 127.1.217.65:37833)
0926 07:04:17.033954 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:17.033966 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:17.034019 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:17.035141 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:17.035144 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:17.035830 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:17.035976 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:17.035984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.037238 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.037239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:17.038072 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:17.038075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:17.038677 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:17.038678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:17.038806 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:17.039452 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:17.039456 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:17.041013 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:17.043118 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043125 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:17.043127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:17.043438 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:17.043699 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:17.043700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:17.043700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:17.043820 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:17.044225 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:17.044231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:17.044367 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 07:04:17.044450 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:04:17.044756 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":148}
W20260926 07:04:17.045951 16756 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
W20260926 07:04:17.046057 16755 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:17.046136 16755 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788; no voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4
I20260926 07:04:17.046270 17140 raft_consensus.cc:2726] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:04:17.617393 17144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.617157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:56647)
0926 07:04:17.617311 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:17.617347 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260926 07:04:17.680958 17145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.680653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:42051)
0926 07:04:17.680888 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:17.680923 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":56,"threads_started":1}
I20260926 07:04:17.802455 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:17.802072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:32847)
0926 07:04:17.802354 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:17.802401 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":64,"threads_started":1}
I20260926 07:04:18.611393 17147 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:18.611491 17147 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:18.611692 17147 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
W20260926 07:04:18.612140 16755 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
W20260926 07:04:18.612218 16756 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:18.612294 16756 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788; no voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4
I20260926 07:04:18.612351 17147 raft_consensus.cc:2726] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:04:18.618007 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:18.617819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:49607)
0926 07:04:18.617923 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:18.617957 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260926 07:04:18.681599 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:18.681420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:35509)
0926 07:04:18.681533 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:18.681564 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 07:04:18.803215 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:18.802901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:39581)
0926 07:04:18.803133 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:18.803171 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":75,"threads_started":1}
I20260926 07:04:19.618718 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:19.618463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:36329)
0926 07:04:19.618621 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:19.618658 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":98,"threads_started":1}
I20260926 07:04:19.682272 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:19.682065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:56371)
0926 07:04:19.682195 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:19.682236 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260926 07:04:19.804010 17153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:19.803783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:44031)
0926 07:04:19.803928 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:19.803965 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 07:04:20.365983 17154 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:20.366094 17154 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:20.366348 17154 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
W20260926 07:04:20.366608 16756 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
W20260926 07:04:20.366673 16755 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:20.366730 16755 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788; no voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4
I20260926 07:04:20.366802 17154 raft_consensus.cc:2726] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:04:20.619448 17155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:20.619245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:46671)
0926 07:04:20.619371 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:20.619402 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260926 07:04:20.682852 17156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:20.682670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:50255)
0926 07:04:20.682788 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:20.682817 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260926 07:04:20.804811 17157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:20.804529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:59981)
0926 07:04:20.804682 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:20.804725 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260926 07:04:21.620308 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:21.620037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:52819)
0926 07:04:21.620215 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:21.620260 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260926 07:04:21.683560 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:21.683311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:51539)
0926 07:04:21.683462 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:21.683519 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260926 07:04:21.805486 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:21.805278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:48889)
0926 07:04:21.805404 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:21.805443 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260926 07:04:22.621179 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:22.620971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:43395)
0926 07:04:22.621097 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:22.621133 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260926 07:04:22.684441 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:22.684031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:49563)
0926 07:04:22.684168 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:22.684211 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260926 07:04:22.732981 17163 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:22.733075 17163 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:22.733232 17163 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
W20260926 07:04:22.733633 16755 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
W20260926 07:04:22.733728 16756 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:22.733804 16756 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788; no voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4
I20260926 07:04:22.733875 17163 raft_consensus.cc:2726] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:04:22.806074 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:22.805871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:47239)
0926 07:04:22.805984 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:22.806016 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260926 07:04:23.621815 17165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:23.621598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:38699)
0926 07:04:23.621738 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:23.621769 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 07:04:23.685070 17166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:23.684887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:50601)
0926 07:04:23.684997 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:23.685029 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260926 07:04:23.806787 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:23.806561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:60113)
0926 07:04:23.806699 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:23.806739 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260926 07:04:24.622499 17168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:24.622270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:57895)
0926 07:04:24.622404 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:24.622435 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
I20260926 07:04:24.685738 17169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:24.685542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:56131)
0926 07:04:24.685660 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:24.685696 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 07:04:24.807452 17170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:24.807243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:34549)
0926 07:04:24.807378 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:24.807409 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260926 07:04:25.623250 17171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:25.623043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:49609)
0926 07:04:25.623166 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:25.623204 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 07:04:25.686367 17172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:25.686178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:57787)
0926 07:04:25.686293 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:25.686328 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
I20260926 07:04:25.808187 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:25.807950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:38981)
0926 07:04:25.808102 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:25.808141 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260926 07:04:26.623991 17174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:26.623788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:49949)
0926 07:04:26.623909 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:26.623944 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260926 07:04:26.686990 17175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:26.686791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:52707)
0926 07:04:26.686913 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:26.686946 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260926 07:04:26.808929 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:26.808687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:46083)
0926 07:04:26.808848 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:26.808886 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
I20260926 07:04:27.624650 17177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:27.624443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:51727)
0926 07:04:27.624570 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:27.624604 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260926 07:04:27.687718 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:27.687467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:49635)
0926 07:04:27.687635 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:27.687675 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":83,"threads_started":1}
I20260926 07:04:27.809526 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:27.809334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:46803)
0926 07:04:27.809450 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:27.809483 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260926 07:04:28.625368 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:28.625144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:34155)
0926 07:04:28.625281 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:28.625321 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260926 07:04:28.688541 17181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:28.688317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:57517)
0926 07:04:28.688461 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:28.688498 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260926 07:04:28.810168 17182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:28.809955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:53675)
0926 07:04:28.810076 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:28.810122 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 07:04:29.626050 17183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:29.625841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:52763)
0926 07:04:29.625963 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:29.626001 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 07:04:29.689275 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:29.689031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:35071)
0926 07:04:29.689191 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:29.689229 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260926 07:04:29.810788 17185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:29.810586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:33075)
0926 07:04:29.810705 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:29.810741 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260926 07:04:30.028012 17186 raft_consensus.cc:482] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:30.028124 17186 raft_consensus.cc:504] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f2ccc9bb9a4708b89508b757c91a23" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 32863 } } peers { permanent_uuid: "cc2a33e80e5a4eb9aa714d6a322f44a4" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 38225 } } peers { permanent_uuid: "7b9c26daf6a84586a83a1a9e7600e788" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41219 } }
I20260926 07:04:30.028302 17186 leader_election.cc:288] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863), cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225)
W20260926 07:04:30.028734 16755 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 33f2ccc9bb9a4708b89508b757c91a23 (127.1.217.67:32863): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
W20260926 07:04:30.028877 16756 leader_election.cc:341] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cc2a33e80e5a4eb9aa714d6a322f44a4 (127.1.217.66:38225): Not found: Tablet not found: 8265dadaac6e4edabf0f3fc0c5727d0c
I20260926 07:04:30.028914 16756 leader_election.cc:302] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7b9c26daf6a84586a83a1a9e7600e788; no voters: 33f2ccc9bb9a4708b89508b757c91a23, cc2a33e80e5a4eb9aa714d6a322f44a4
I20260926 07:04:30.029004 17186 raft_consensus.cc:2726] T 8265dadaac6e4edabf0f3fc0c5727d0c P 7b9c26daf6a84586a83a1a9e7600e788 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:04:30.626819 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:30.626564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.65:44325)
0926 07:04:30.626688 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:30.626763 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 07:04:30.690150 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:30.689853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.66:38927)
0926 07:04:30.690029 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:30.690066 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260926 07:04:30.810041 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16708
I20260926 07:04:30.811501 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:30.811284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:43825 (local address 127.1.217.67:54431)
0926 07:04:30.811410 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:30.811443 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:43825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260926 07:04:30.814813 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 16872
I20260926 07:04:30.819109 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17004
2026-09-26T07:04:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18194 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:04:30 dist-test-slave-t1mc krb5kdc[17192](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:04:30 dist-test-slave-t1mc krb5kdc[17192](info): set up 2 sockets
Sep 26 07:04:30 dist-test-slave-t1mc krb5kdc[17192](info): commencing operation
krb5kdc: starting...
W20260926 07:04:31.864943 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:04:31 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:04:31.983765 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:32.066866 17205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:32.067107 17205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:32.067132 17205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:32.068383 17205 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:04:32.068428 17205 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:04:32.068441 17205 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:04:32.068454 17205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:32.068506 17205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:32.068526 17205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:32.068550 17205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:32.068568 17205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:32.070187 17205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:32.070466 17205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:32.070717 17205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:32.076349 17211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:32.076316 17205 server_base.cc:1034] running on GCE node
W20260926 07:04:32.076359 17210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.076577 17213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:32.077018 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:32.077329 17205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:32.078485 17205 hybrid_clock.cc:648] HybridClock initialized: now 1790406272078475 us; error 32 us; skew 500 ppm
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:32.081048 17205 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:32.081553 17205 webserver.cc:460] Webserver started at http://127.1.217.126:40985/ using document root <none> and password file <none>
I20260926 07:04:32.081790 17205 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:32.081861 17205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:32.082017 17205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:32.083117 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "7f8061e5f4404117b4e355486a136e0c"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "e6e6281ba8ee154800589dc9c67f7e0c"
server_key_iv: "29497dc0a74c432f92f6f1c1f19d5be7"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.083489 17205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "7f8061e5f4404117b4e355486a136e0c"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "e6e6281ba8ee154800589dc9c67f7e0c"
server_key_iv: "29497dc0a74c432f92f6f1c1f19d5be7"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.084903 17205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:04:32.085646 17220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.085832 17205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:32.085934 17205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "7f8061e5f4404117b4e355486a136e0c"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "e6e6281ba8ee154800589dc9c67f7e0c"
server_key_iv: "29497dc0a74c432f92f6f1c1f19d5be7"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.086024 17205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:32.126832 17205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:32.128760 17205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:32.128991 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:32.132962 17205 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33381
I20260926 07:04:32.132998 17272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33381 every 8 connection(s)
I20260926 07:04:32.133358 17205 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:32.134210 17273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:32.136488 17273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap starting.
I20260926 07:04:32.137135 17273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:32.137447 17273 log.cc:824] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:32.138197 17273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: No bootstrap required, opened a new log
I20260926 07:04:32.138746 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17205
I20260926 07:04:32.138833 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:04:32.139024 1893 external_mini_cluster.cc:1428] Setting key cccc023182c43f622a72b7e3ec555426
I20260926 07:04:32.140130 17273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:32.140290 17273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:32.140352 17273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Initialized, Role: FOLLOWER
I20260926 07:04:32.140515 17273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:32.140604 17273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:32.140653 17273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:32.140708 17273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:32.141520 17273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:32.141650 17273 leader_election.cc:302] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f8061e5f4404117b4e355486a136e0c; no voters:
I20260926 07:04:32.141825 17273 leader_election.cc:288] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:04:32.141873 17278 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:32.142019 17278 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 LEADER]: Becoming Leader. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Running, Role: LEADER
I20260926 07:04:32.142091 17273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:32.142166 17278 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:32.142529 17278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f8061e5f4404117b4e355486a136e0c. Latest consensus state: current_term: 1 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:04:32.142670 17278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:32.142851 17279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:04:32.142920 17279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:32.143607 17288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:32.144069 17288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:32.145684 17288 catalog_manager.cc:1347] Generated new cluster ID: 96f4f359d29f474da6be4bacb85b4a33
I20260926 07:04:32.145740 17288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:32.151117 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.139595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.1.217.126:33381)
0926 07:04:32.139790 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.139800 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.139837 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:04:32.140135 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.140143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.140205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.140422 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:32.141160 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.142440 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.143522 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.144146 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.144580 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.147246 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:32.147249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:32.147250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:32.147261 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:32.149267 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.149655 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.149658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.149658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.149765 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.149939 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.149939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.149940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.150215 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:32.150707 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.150774 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.150897 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260926 07:04:32.176241 17288 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:04:32.176863 17288 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:32.187623 17288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Generated new TSK 0
I20260926 07:04:32.187850 17288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:04:32.220295 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:32.301548 17299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:32.301744 17299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:32.301767 17299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:32.302903 17299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:32.302943 17299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:32.302969 17299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:32.304414 17299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:32.304600 17299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:32.304880 17299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:32.309701 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.309710 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.309756 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:32.310056 17299 server_base.cc:1034] running on GCE node
I20260926 07:04:32.310283 17299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:32.310519 17299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:32.311717 17299 hybrid_clock.cc:648] HybridClock initialized: now 1790406272311699 us; error 27 us; skew 500 ppm
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:32.318383 17299 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:32.318939 17299 webserver.cc:460] Webserver started at http://127.1.217.65:45185/ using document root <none> and password file <none>
I20260926 07:04:32.319195 17299 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:32.319269 17299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:32.319425 17299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:32.320495 17299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "b9bd22788b7b410fa1341267b58b8718"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "a7bde99fd74ec4c720ced85c5ae0bd7a"
server_key_iv: "fb291a4a929ed10b2d01f5f83bef0438"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.320933 17299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "b9bd22788b7b410fa1341267b58b8718"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "a7bde99fd74ec4c720ced85c5ae0bd7a"
server_key_iv: "fb291a4a929ed10b2d01f5f83bef0438"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.322295 17299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:04:32.323077 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.323259 17299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.323359 17299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "b9bd22788b7b410fa1341267b58b8718"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "a7bde99fd74ec4c720ced85c5ae0bd7a"
server_key_iv: "fb291a4a929ed10b2d01f5f83bef0438"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.323447 17299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:32.343743 17299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:32.345795 17299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:32.345960 17299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:32.346207 17299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:32.346539 17299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:32.346596 17299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.346633 17299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:32.346661 17299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.352576 17299 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:34389
I20260926 07:04:32.352619 17427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:34389 every 8 connection(s)
I20260926 07:04:32.352984 17299 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:32.355485 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17299
I20260926 07:04:32.355623 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:04:32.355821 1893 external_mini_cluster.cc:1428] Setting key 8d97c3b5fd64eeed0ae4f27670ca9750
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:32.362910 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.353717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:50851 (local address 127.1.217.126:33381)
0926 07:04:32.353790 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.353791 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.354011 (+ 220us) server_negotiation.cc:407] Connection header received
0926 07:04:32.354485 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.354486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.354495 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.354568 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:32.355502 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.356157 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.357299 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.357526 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.357615 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.359810 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:32.359812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:32.359813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:32.359823 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:32.361548 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.361924 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.361925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.361925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.361989 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.362237 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.362238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.362239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.362382 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:32.362454 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.362693 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.362789 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:04:32.362916 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.353699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:50851)
0926 07:04:32.353873 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.353917 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.354406 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.354624 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.354627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.355190 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:32.355402 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.355407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.356252 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.356253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.357208 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.357211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.357626 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.357627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.357762 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.358316 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:32.358326 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:32.359696 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:32.361630 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.361636 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.361638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.361849 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.362060 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.362060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.362060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.362160 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.362446 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:32.362450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:32.362561 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.362641 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.362758 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260926 07:04:32.363564 17428 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:32.363660 17428 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:32.363857 17428 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
I20260926 07:04:32.364465 17237 ts_manager.cc:194] Registered new tserver with Master: b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:32.365171 17237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:50851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:04:32.383066 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:32.477983 17433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:32.478423 17433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:32.478497 17433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:32.480803 17433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:32.480866 17433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:32.480929 17433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:32.483675 17433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:32.483968 17433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:32.484256 17433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:32.490675 17441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.490671 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.490671 17439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:32.490968 17433 server_base.cc:1034] running on GCE node
I20260926 07:04:32.491226 17433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:32.491480 17433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:32.492648 17433 hybrid_clock.cc:648] HybridClock initialized: now 1790406272492636 us; error 30 us; skew 500 ppm
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:32.495296 17433 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:32.495785 17433 webserver.cc:460] Webserver started at http://127.1.217.66:36633/ using document root <none> and password file <none>
I20260926 07:04:32.496014 17433 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:32.496084 17433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:32.496244 17433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:32.497445 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "75bf51ae30944ac1831277b3b7aee23f"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "15ddf1637c930dc04786016d886768d7"
server_key_iv: "fcf3adbe84fae57d4e427df766305a6a"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.497814 17433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "75bf51ae30944ac1831277b3b7aee23f"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "15ddf1637c930dc04786016d886768d7"
server_key_iv: "fcf3adbe84fae57d4e427df766305a6a"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.499121 17433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:04:32.499843 17448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.500011 17433 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.500116 17433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "75bf51ae30944ac1831277b3b7aee23f"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "15ddf1637c930dc04786016d886768d7"
server_key_iv: "fcf3adbe84fae57d4e427df766305a6a"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.500216 17433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:32.516429 17433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:32.518831 17433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:32.519003 17433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:32.519289 17433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:32.519676 17433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:32.519726 17433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.519767 17433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:32.519795 17433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.525701 17433 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:36263
I20260926 07:04:32.525748 17561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:36263 every 8 connection(s)
I20260926 07:04:32.526103 17433 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:32.527904 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17433
I20260926 07:04:32.528031 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:04:32.528261 1893 external_mini_cluster.cc:1428] Setting key 3ff7db4956b927ea6dac2b47a24d42fd
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
I20260926 07:04:32.536477 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.526844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:53591 (local address 127.1.217.126:33381)
0926 07:04:32.526916 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.526920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.527180 (+ 260us) server_negotiation.cc:407] Connection header received
0926 07:04:32.527703 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.527704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.527732 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.527814 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:32.529079 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.530119 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.531135 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.531353 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.531442 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.533723 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:32.533725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:32.533725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:32.533733 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:32.535072 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.535501 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.535502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.535503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.535564 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.535826 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.535827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.535827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.535946 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:32.536022 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.536297 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.536383 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:04:32.536687 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.526837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:53591)
0926 07:04:32.527027 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.527080 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.527616 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.527869 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.527872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.528653 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:32.528970 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.528978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.530226 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.530227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.531022 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.531024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.531451 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.531453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.531591 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.532141 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:32.532149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:32.533613 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:32.535170 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.535177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.535179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.535409 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.535632 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.535633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.535633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.535752 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.536013 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:32.536020 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:32.536163 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.536278 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.536438 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":108,"threads_started":1}
I20260926 07:04:32.537220 17562 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:32.537366 17562 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:32.537564 17562 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
I20260926 07:04:32.537917 17237 ts_manager.cc:194] Registered new tserver with Master: 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
I20260926 07:04:32.538276 17237 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:53591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:04:32.555073 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:32.637756 17567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:32.637964 17567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:32.638026 17567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:32.639228 17567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:32.639269 17567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:32.639329 17567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:32.640815 17567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:32.641027 17567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:32.641217 17567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:32.646157 17572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.646135 17575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:32.646126 17573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:32.646535 17567 server_base.cc:1034] running on GCE node
I20260926 07:04:32.646744 17567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:32.646943 17567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:32.648085 17567 hybrid_clock.cc:648] HybridClock initialized: now 1790406272648073 us; error 28 us; skew 500 ppm
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:32.650663 17567 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:32.651131 17567 webserver.cc:460] Webserver started at http://127.1.217.67:44439/ using document root <none> and password file <none>
I20260926 07:04:32.651356 17567 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:32.651422 17567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:32.651567 17567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:04:32.652597 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "40f33cdf1cb24d319b2d528378de6d16"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "921e946990c43043a397e68bf240189f"
server_key_iv: "a475d1a777514891e325a6153d32f708"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.652985 17567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "40f33cdf1cb24d319b2d528378de6d16"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "921e946990c43043a397e68bf240189f"
server_key_iv: "a475d1a777514891e325a6153d32f708"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.654366 17567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:32.655022 17582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.655194 17567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:04:32.655287 17567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "40f33cdf1cb24d319b2d528378de6d16"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "921e946990c43043a397e68bf240189f"
server_key_iv: "a475d1a777514891e325a6153d32f708"
server_key_version: "encryptionkey@0"
I20260926 07:04:32.655362 17567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:32.662220 17567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:32.664367 17567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:32.664518 17567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:32.664729 17567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:32.665032 17567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:32.665081 17567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.665122 17567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:32.665146 17567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:32.670979 17567 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:36139
I20260926 07:04:32.671021 17695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:36139 every 8 connection(s)
I20260926 07:04:32.671329 17567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:04:32 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406272, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:32.680033 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17567
I20260926 07:04:32.680173 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:04:32.680364 1893 external_mini_cluster.cc:1428] Setting key b834be43baee1a6989bdcca1d86a32b5
I20260926 07:04:32.680704 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.672123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:47401 (local address 127.1.217.126:33381)
0926 07:04:32.672212 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.672215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.672412 (+ 197us) server_negotiation.cc:407] Connection header received
0926 07:04:32.673007 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.673009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.673021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.673066 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:32.674024 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.674526 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.675553 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.675764 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.675848 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.678074 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:32.678075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:32.678076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:32.678083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:32.679268 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.679625 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.679626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.679626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.679676 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.679889 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.679890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.679890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.680034 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:32.680109 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.680521 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.680592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:04:32.680910 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.672083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:47401)
0926 07:04:32.672260 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.672310 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.672904 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.673144 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.673148 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.673720 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:32.673912 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.673917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.674619 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.674620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.675455 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.675457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.675859 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.675860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.675995 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.676534 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:32.676542 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:32.677987 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:32.679339 (+ 1352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.679344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.679346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.679561 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.679733 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:32.679733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:32.679734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:32.679827 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:32.680166 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:32.680175 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:32.680326 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.680462 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.680646 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260926 07:04:32.681408 17696 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:32.681512 17696 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:32.681699 17696 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
I20260926 07:04:32.682055 17236 ts_manager.cc:194] Registered new tserver with Master: 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139)
I20260926 07:04:32.682447 17236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:47401
I20260926 07:04:32.692304 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:32.699586 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.693652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.1.217.126:33381)
0926 07:04:32.693734 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.693736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.693842 (+ 106us) server_negotiation.cc:407] Connection header received
0926 07:04:32.693941 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.693942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.693952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.694006 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:32.694545 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.695032 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.695780 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.695981 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.696070 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.697254 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:32.697256 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:32.697256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:32.697263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:32.698404 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.698753 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.698754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.698754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.698815 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:32.699022 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:32.699023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:32.699023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:32.699179 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:32.699295 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.699404 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.699483 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:04:32.701213 17236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:04:32.702754 17236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:04:32.709465 17709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:36139 (local address 127.0.0.1:51396)
0926 07:04:32.705473 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.705485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.705539 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.705812 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.705816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.705818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:32.705891 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.705892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.706932 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.706933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.708207 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.708210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.709143 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.709144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.709199 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.709210 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.709310 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.709374 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":92,"threads_started":1}
I20260926 07:04:32.709512 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.1.217.65:34389)
0926 07:04:32.705452 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.705458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.705685 (+ 227us) server_negotiation.cc:407] Connection header received
0926 07:04:32.705737 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.705740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.705755 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.705851 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:32.706069 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.707200 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.708473 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.709184 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.709249 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.709295 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.709394 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.709431 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 07:04:32.709730 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.1.217.67:36139)
0926 07:04:32.705548 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.705552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.705563 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:04:32.705608 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.705610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.705628 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.705707 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:32.705976 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.706832 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.708307 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.709067 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.709145 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.709206 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.709585 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.709642 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
I20260926 07:04:32.709476 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.0.0.1:44306)
0926 07:04:32.705621 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.705629 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.705678 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.705921 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.705922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.705924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:32.705986 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.705987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.707295 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.707296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.708398 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.708399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.709254 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.709255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.709329 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.709332 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.709382 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.709417 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":43,"thread_start_us":77,"threads_started":1}
I20260926 07:04:32.710536 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.0.0.1:53048)
0926 07:04:32.705289 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.705332 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.705399 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.705923 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.705924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.705925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:32.705986 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.705987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.707303 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.707304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.708227 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.708229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.710296 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.710297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.710383 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.710386 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.710428 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.710474 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":38,"thread_start_us":60,"threads_started":1}
I20260926 07:04:32.710450 17630 tablet_service.cc:1467] Processing CreateTablet for tablet a1e4b56524314cfbad25d274e244352e (DEFAULT_TABLE table=test-table [id=9c68e63c15654163a26f0305ccb56df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:32.710805 17630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e4b56524314cfbad25d274e244352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:32.710889 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.705092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.1.217.66:36263)
0926 07:04:32.705482 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.705487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.705496 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:32.705532 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.705534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.705557 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.705681 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:32.706072 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.707185 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.709241 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.709999 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.710225 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.710742 (+ 517us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.710768 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.710811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":99,"threads_started":1}
I20260926 07:04:32.710810 17362 tablet_service.cc:1467] Processing CreateTablet for tablet a1e4b56524314cfbad25d274e244352e (DEFAULT_TABLE table=test-table [id=9c68e63c15654163a26f0305ccb56df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:32.711140 17362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e4b56524314cfbad25d274e244352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:32.711483 17496 tablet_service.cc:1467] Processing CreateTablet for tablet a1e4b56524314cfbad25d274e244352e (DEFAULT_TABLE table=test-table [id=9c68e63c15654163a26f0305ccb56df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:32.711792 17496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1e4b56524314cfbad25d274e244352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:32.713526 17713 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Bootstrap starting.
I20260926 07:04:32.714149 17713 tablet_bootstrap.cc:654] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:32.714509 17714 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Bootstrap starting.
I20260926 07:04:32.714789 17713 log.cc:824] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:32.715209 17714 tablet_bootstrap.cc:654] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:32.715189 17716 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Bootstrap starting.
I20260926 07:04:32.715608 17713 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: No bootstrap required, opened a new log
I20260926 07:04:32.715695 17713 ts_tablet_manager.cc:1397] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:32.715792 17714 log.cc:824] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:32.715850 17716 tablet_bootstrap.cc:654] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:32.716492 17716 log.cc:824] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:32.716575 17714 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: No bootstrap required, opened a new log
I20260926 07:04:32.716629 17714 ts_tablet_manager.cc:1397] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:32.717299 17716 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: No bootstrap required, opened a new log
I20260926 07:04:32.717370 17716 ts_tablet_manager.cc:1397] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:32.717876 17713 raft_consensus.cc:348] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.718053 17713 raft_consensus.cc:374] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:32.718091 17713 raft_consensus.cc:729] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f33cdf1cb24d319b2d528378de6d16, State: Initialized, Role: FOLLOWER
I20260926 07:04:32.718204 17713 consensus_queue.cc:260] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.718221 17714 raft_consensus.cc:348] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.718416 17714 raft_consensus.cc:374] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:32.718451 17714 raft_consensus.cc:729] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bd22788b7b410fa1341267b58b8718, State: Initialized, Role: FOLLOWER
I20260926 07:04:32.718479 17713 ts_tablet_manager.cc:1428] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:04:32.718596 17696 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
I20260926 07:04:32.718602 17714 consensus_queue.cc:260] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.718910 17428 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
I20260926 07:04:32.718910 17714 ts_tablet_manager.cc:1428] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:32.719339 17716 raft_consensus.cc:348] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.719517 17716 raft_consensus.cc:374] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:32.719565 17716 raft_consensus.cc:729] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bf51ae30944ac1831277b3b7aee23f, State: Initialized, Role: FOLLOWER
I20260926 07:04:32.719677 17716 consensus_queue.cc:260] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.719957 17716 ts_tablet_manager.cc:1428] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:04:32.719974 17562 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
W20260926 07:04:32.776758 17563 tablet.cc:2369] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:32.805724 17719 raft_consensus.cc:482] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:32.805857 17719 raft_consensus.cc:504] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.806205 17719 leader_election.cc:288] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263), b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:32.809161 17698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.806383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.1.217.67:49491)
0926 07:04:32.806437 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.806448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.806489 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.806665 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.806666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.806669 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:32.806743 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.806745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.807460 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.807461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.808309 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.808311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.808882 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.808964 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.808969 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.809028 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.809062 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":67}
I20260926 07:04:32.809180 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.806457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:49491 (local address 127.1.217.65:34389)
0926 07:04:32.806498 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.806500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.806528 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:04:32.806556 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.806557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.806567 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.806635 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:32.806833 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.807370 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808408 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.808803 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808859 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.808865 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.809036 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.809080 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:32.809240 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.806483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:41391 (local address 127.1.217.66:36263)
0926 07:04:32.806530 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.806532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.806661 (+ 129us) server_negotiation.cc:407] Connection header received
0926 07:04:32.806734 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.806735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.806756 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.806808 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:32.806971 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.807440 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808444 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.808885 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808957 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.808964 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.809133 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.809169 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:04:32.809301 17722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.806393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.1.217.67:41391)
0926 07:04:32.806591 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.806600 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:32.806663 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:32.806834 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:32.806834 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:32.806836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:32.806900 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.806901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.807539 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.807540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.808325 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:32.808327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.808969 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.808970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:32.809052 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.809055 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:04:32.809110 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 07:04:32.809144 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":16,"thread_start_us":122,"threads_started":1}
I20260926 07:04:32.809568 17382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e4b56524314cfbad25d274e244352e" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bd22788b7b410fa1341267b58b8718" is_pre_election: true
I20260926 07:04:32.809569 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e4b56524314cfbad25d274e244352e" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf51ae30944ac1831277b3b7aee23f" is_pre_election: true
I20260926 07:04:32.809728 17516 raft_consensus.cc:2454] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 0.
I20260926 07:04:32.809728 17382 raft_consensus.cc:2454] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 0.
I20260926 07:04:32.809966 17586 leader_election.cc:302] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f33cdf1cb24d319b2d528378de6d16, b9bd22788b7b410fa1341267b58b8718; no voters:
I20260926 07:04:32.810124 17719 raft_consensus.cc:2781] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:32.810209 17719 raft_consensus.cc:482] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:32.810252 17719 raft_consensus.cc:3037] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:32.810976 17719 raft_consensus.cc:504] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.811131 17719 leader_election.cc:288] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 election: Requested vote from peers 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263), b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:32.811278 17382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e4b56524314cfbad25d274e244352e" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bd22788b7b410fa1341267b58b8718"
I20260926 07:04:32.811304 17516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1e4b56524314cfbad25d274e244352e" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf51ae30944ac1831277b3b7aee23f"
I20260926 07:04:32.811376 17516 raft_consensus.cc:3037] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:32.811383 17382 raft_consensus.cc:3037] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:32.812104 17516 raft_consensus.cc:2454] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 1.
I20260926 07:04:32.812280 17382 raft_consensus.cc:2454] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 1.
I20260926 07:04:32.812292 17583 leader_election.cc:302] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f; no voters:
I20260926 07:04:32.812492 17719 raft_consensus.cc:2781] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:32.812640 17719 raft_consensus.cc:686] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 1 LEADER]: Becoming Leader. State: Replica: 40f33cdf1cb24d319b2d528378de6d16, State: Running, Role: LEADER
I20260926 07:04:32.812711 17719 consensus_queue.cc:237] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:32.813621 17237 catalog_manager.cc:5495] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67). New cstate: current_term: 1 leader_uuid: "40f33cdf1cb24d319b2d528378de6d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } health_report { overall_health: UNKNOWN } } }
I20260926 07:04:32.823948 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.821807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.1.217.67:36139)
0926 07:04:32.821859 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:32.821860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:32.821869 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:04:32.821951 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:32.821952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:32.821967 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:32.822024 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:32.822133 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.822588 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.823300 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:32.823486 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:32.823540 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:32.823672 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:32.823768 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:32.823828 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:32.823858 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 07:04:32.823875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 07:04:32.825767 17382 raft_consensus.cc:1261] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Refusing update from remote peer 40f33cdf1cb24d319b2d528378de6d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:32.825793 17516 raft_consensus.cc:1261] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Refusing update from remote peer 40f33cdf1cb24d319b2d528378de6d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:32.826115 17723 consensus_queue.cc:1035] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:32.826253 17719 consensus_queue.cc:1035] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:32.828783 17726 mvcc.cc:205] Tried to move back new op lower bound from 7333504093491347456 to 7333504093441699840. Current Snapshot: MvccSnapshot[applied={T|T < 7333504093491347456}]
I20260926 07:04:32.829777 17727 mvcc.cc:205] Tried to move back new op lower bound from 7333504093491347456 to 7333504093441699840. Current Snapshot: MvccSnapshot[applied={T|T < 7333504093491347456 or (T in {7333504093491347456})}]
I20260926 07:04:32.830969 17236 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:32.831091 17236 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:32.831856 17236 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Sending DeleteTablet for 3 replicas of tablet a1e4b56524314cfbad25d274e244352e
I20260926 07:04:32.832208 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17299
I20260926 07:04:32.832190 17496 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e4b56524314cfbad25d274e244352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:32 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53048
I20260926 07:04:32.832211 17362 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e4b56524314cfbad25d274e244352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:32 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:44306
I20260926 07:04:32.832291 17630 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e4b56524314cfbad25d274e244352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:32 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:51396
I20260926 07:04:32.836902 17737 tablet_replica.cc:330] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: stopping tablet replica
I20260926 07:04:32.837025 17737 raft_consensus.cc:2229] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:04:32.837159 17737 raft_consensus.cc:2258] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:32.837250 17736 tablet_replica.cc:330] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: stopping tablet replica
I20260926 07:04:32.837352 17736 raft_consensus.cc:2229] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:32.837451 17736 raft_consensus.cc:2258] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:32.837960 17736 ts_tablet_manager.cc:1905] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:32.837960 17737 ts_tablet_manager.cc:1905] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 07:04:32.838626 17224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:34389 (error 108)
I20260926 07:04:32.839177 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17433
I20260926 07:04:32.839836 17737 ts_tablet_manager.cc:1918] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:32.839974 17737 log.cc:1196] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/a1e4b56524314cfbad25d274e244352e
I20260926 07:04:32.840301 17737 ts_tablet_manager.cc:1939] T a1e4b56524314cfbad25d274e244352e P 40f33cdf1cb24d319b2d528378de6d16: Deleting consensus metadata
I20260926 07:04:32.841045 17222 catalog_manager.cc:4841] TS 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): tablet a1e4b56524314cfbad25d274e244352e (table test-table [id=9c68e63c15654163a26f0305ccb56df5]) successfully deleted
I20260926 07:04:32.841060 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.840909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.0.0.1:44312)
0926 07:04:32.840972 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.841017 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 07:04:32.842092 17224 catalog_manager.cc:4584] TS b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1e4b56524314cfbad25d274e244352e: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34389: connect: Connection refused (error 111)
I20260926 07:04:32.845014 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17567
I20260926 07:04:32.845069 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:32.844960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.0.0.1:53062)
0926 07:04:32.845004 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:32.845033 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.66:36263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 07:04:32.850525 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17205
I20260926 07:04:52.855970 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--webserver_interface=127.1.217.126
--webserver_port=40985
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:04:52.937394 17739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:52.937608 17739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:52.937631 17739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:52.938819 17739 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:04:52.938859 17739 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:04:52.938874 17739 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:04:52.938915 17739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:04:52.938948 17739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:52.938967 17739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:52.938993 17739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:04:52.939033 17739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:04:52.940593 17739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=40985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:52.940894 17739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:52.941138 17739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:52.949502 17746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:52.949550 17745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:52.949512 17748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:52.949754 17739 server_base.cc:1034] running on GCE node
I20260926 07:04:52.950026 17739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:52.950330 17739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:52.951488 17739 hybrid_clock.cc:648] HybridClock initialized: now 1790406292951468 us; error 38 us; skew 500 ppm
Sep 26 07:04:52 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406292, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:52.954100 17739 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:04:52.954597 17739 webserver.cc:460] Webserver started at http://127.1.217.126:40985/ using document root <none> and password file <none>
I20260926 07:04:52.954845 17739 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:52.954932 17739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:52.956271 17739 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:52.956965 17755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:52.957161 17739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:52.957283 17739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "7f8061e5f4404117b4e355486a136e0c"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "e6e6281ba8ee154800589dc9c67f7e0c"
server_key_iv: "29497dc0a74c432f92f6f1c1f19d5be7"
server_key_version: "encryptionkey@0"
I20260926 07:04:52.957700 17739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:52.977185 17739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:52.979581 17739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:52.979750 17739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:52.983898 17739 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33381
I20260926 07:04:52.983935 17807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33381 every 8 connection(s)
I20260926 07:04:52.984344 17739 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:04:52.985867 17808 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:04:52.986444 17808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap starting.
I20260926 07:04:52.991147 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17739
I20260926 07:04:52.991449 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:34389
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=45185
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:52.994887 17808 log.cc:824] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:52.996749 17808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:52.997120 17808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap complete.
I20260926 07:04:52.999356 17808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:52.999584 17808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Initialized, Role: FOLLOWER
I20260926 07:04:52.999747 17808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:52.999814 17808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:04:52.999877 17808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:04:52.999933 17808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 1 FOLLOWER]: Advancing to term 2
I20260926 07:04:53.001046 17808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:53.001134 17808 leader_election.cc:302] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f8061e5f4404117b4e355486a136e0c; no voters:
I20260926 07:04:53.001348 17808 leader_election.cc:288] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 07:04:53.001380 17812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 07:04:53.001497 17812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 LEADER]: Becoming Leader. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Running, Role: LEADER
I20260926 07:04:53.001652 17808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:04:53.001639 17812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:04:53.001959 17814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f8061e5f4404117b4e355486a136e0c. Latest consensus state: current_term: 2 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:04:53.002094 17814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:53.001962 17813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:04:53.002357 17813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:04:53.002305 17818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:04:53.003268 17818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:04:53.003407 17818 catalog_manager.cc:1259] Loaded cluster ID: 96f4f359d29f474da6be4bacb85b4a33
I20260926 07:04:53.003652 17818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:04:53.004437 17818 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:04:53.004567 17818 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Loaded TSK: 0
I20260926 07:04:53.013734 17818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Generated new TSK 1
I20260926 07:04:53.013824 17818 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:04:53.082180 17810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:53.082423 17810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:53.082463 17810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:53.083601 17810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:53.083668 17810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:53.083715 17810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:04:53.085198 17810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=45185
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:53.085460 17810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:53.085685 17810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:53.090647 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:53.090639 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:53.090945 17810 server_base.cc:1034] running on GCE node
W20260926 07:04:53.090641 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:53.091187 17810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:53.091480 17810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:53.092645 17810 hybrid_clock.cc:648] HybridClock initialized: now 1790406293092623 us; error 39 us; skew 500 ppm
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:53.095242 17810 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:04:53.095669 17810 webserver.cc:460] Webserver started at http://127.1.217.65:45185/ using document root <none> and password file <none>
I20260926 07:04:53.095899 17810 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:53.095955 17810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:53.097500 17810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 07:04:53.098234 17845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.098436 17810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:53.098538 17810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "b9bd22788b7b410fa1341267b58b8718"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "a7bde99fd74ec4c720ced85c5ae0bd7a"
server_key_iv: "fb291a4a929ed10b2d01f5f83bef0438"
server_key_version: "encryptionkey@0"
I20260926 07:04:53.098898 17810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:53.109951 17810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:53.112232 17810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:53.112399 17810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:53.112663 17810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:53.113159 17852 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:04:53.114590 17810 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:04:53.114646 17810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:04:53.114671 17810 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:04:53.115332 17810 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:04:53.115376 17810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:04:53.115425 17852 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Bootstrap starting.
I20260926 07:04:53.123114 17810 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:34389
I20260926 07:04:53.123275 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:34389 every 8 connection(s)
I20260926 07:04:53.123662 17810 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:04:53.126770 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17810
I20260926 07:04:53.127074 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:36263
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=36633
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:04:53.127583 17852 log.cc:824] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:53.130093 17852 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:04:53.130379 17852 tablet_bootstrap.cc:492] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Bootstrap complete.
I20260926 07:04:53.131928 17852 ts_tablet_manager.cc:1397] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:53.133320 17852 raft_consensus.cc:348] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.133694 17852 raft_consensus.cc:729] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bd22788b7b410fa1341267b58b8718, State: Initialized, Role: FOLLOWER
I20260926 07:04:53.134176 17852 consensus_queue.cc:260] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.134773 17852 ts_tablet_manager.cc:1428] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:04:53.137147 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.124408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:35067)
0926 07:04:53.124611 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.124655 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.125476 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.126020 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.126026 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.126446 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:53.126792 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.126801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.128552 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.128553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.129803 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.129805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.130678 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.130679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.130832 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.131439 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:53.131449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:53.133331 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:53.135499 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.135506 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.135508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.135736 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.136013 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.136014 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.136014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.136128 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.136507 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:53.136514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:53.136655 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.136758 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.136947 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":117,"threads_started":1}
I20260926 07:04:53.137154 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.124509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:35067 (local address 127.1.217.126:33381)
0926 07:04:53.124708 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.124717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.124757 (+ 40us) server_negotiation.cc:407] Connection header received
0926 07:04:53.125723 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.125731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.125789 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.125962 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.126956 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.128405 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.129878 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.130566 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.130681 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.133450 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.133454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.133454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.133466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.135382 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.135837 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.135842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.135842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.135934 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.136206 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.136207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.136207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.136422 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.136515 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.136862 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.136985 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260926 07:04:53.137763 17960 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:53.137878 17960 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:53.138054 17960 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
I20260926 07:04:53.138813 17772 ts_manager.cc:194] Registered new tserver with Master: b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:53.139932 17772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:35067
I20260926 07:04:53.141470 17960 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406292, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.65@KRBTEST.COM
I20260926 07:04:53.148444 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.139379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.0.0.1:53706)
0926 07:04:53.139617 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.139647 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.139704 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.140588 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.140594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.140952 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:53.140998 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.141000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.142468 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.142469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.143088 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.143090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.143584 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.143585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.143662 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.144163 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:53.144170 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:53.145552 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:53.147286 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.147292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.147294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.147489 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.147725 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.147726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.147726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.147793 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.148067 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:53.148074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:53.148125 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.148234 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.148350 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":73,"threads_started":1}
I20260926 07:04:53.148443 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.140243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.1.217.65:34389)
0926 07:04:53.140416 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.140421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.140431 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:04:53.140461 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.140463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.140478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.140545 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.141341 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.142387 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.143180 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.143485 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.143587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.145637 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.145645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.145645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.145652 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.147176 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.147578 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.147581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.147581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.147643 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.147859 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.147860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.147860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.147990 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.148095 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.148268 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.148348 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260926 07:04:53.149179 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet a1e4b56524314cfbad25d274e244352e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:32 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53706
I20260926 07:04:53.149681 17969 tablet_replica.cc:330] stopping tablet replica
I20260926 07:04:53.149785 17969 raft_consensus.cc:2229] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:53.149852 17969 raft_consensus.cc:2258] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:53.150130 17969 ts_tablet_manager.cc:1905] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:53.152267 17969 ts_tablet_manager.cc:1918] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:53.152326 17969 log.cc:1196] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal/wals/a1e4b56524314cfbad25d274e244352e
I20260926 07:04:53.152587 17969 ts_tablet_manager.cc:1939] T a1e4b56524314cfbad25d274e244352e P b9bd22788b7b410fa1341267b58b8718: Deleting consensus metadata
I20260926 07:04:53.153105 17759 catalog_manager.cc:4841] TS b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389): tablet a1e4b56524314cfbad25d274e244352e (table test-table [id=9c68e63c15654163a26f0305ccb56df5]) successfully deleted
W20260926 07:04:53.215695 17965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:53.215931 17965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:53.215978 17965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:53.217154 17965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:53.217224 17965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:53.217271 17965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:04:53.218760 17965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:36263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=36633
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:53.219060 17965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:53.219306 17965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:53.224195 17975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:53.224202 17974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:53.224232 17977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:53.224469 17965 server_base.cc:1034] running on GCE node
I20260926 07:04:53.224704 17965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:53.224977 17965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:53.226122 17965 hybrid_clock.cc:648] HybridClock initialized: now 1790406293226099 us; error 38 us; skew 500 ppm
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:53.228641 17965 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:04:53.229117 17965 webserver.cc:460] Webserver started at http://127.1.217.66:36633/ using document root <none> and password file <none>
I20260926 07:04:53.229311 17965 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:53.229362 17965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:53.230705 17965 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:04:53.231329 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.231508 17965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:04:53.231609 17965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "75bf51ae30944ac1831277b3b7aee23f"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "15ddf1637c930dc04786016d886768d7"
server_key_iv: "fcf3adbe84fae57d4e427df766305a6a"
server_key_version: "encryptionkey@0"
I20260926 07:04:53.231956 17965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:53.259341 17965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:53.261863 17965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:53.262036 17965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:53.262280 17965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:53.262713 17991 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 07:04:53.264045 17991 ts_tablet_manager.cc:1852] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 07:04:53.264302 17991 ts_tablet_manager.cc:1905] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:53.266523 17991 ts_tablet_manager.cc:1918] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:53.266611 17991 log.cc:1196] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/a1e4b56524314cfbad25d274e244352e
I20260926 07:04:53.266820 17991 ts_tablet_manager.cc:1939] T a1e4b56524314cfbad25d274e244352e P 75bf51ae30944ac1831277b3b7aee23f: Deleting consensus metadata
I20260926 07:04:53.266984 17965 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 07:04:53.267053 17965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 07:04:53.267102 17965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:53.267128 17965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.273001 17965 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:36263
I20260926 07:04:53.273059 18098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:36263 every 8 connection(s)
I20260926 07:04:53.273348 17965 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:04:53.274382 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 17965
I20260926 07:04:53.274691 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:36139
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=44439
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:53.283463 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.274039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:40087)
0926 07:04:53.274231 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.274273 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.274921 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.275193 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.275197 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.275446 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:53.275653 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.275657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.276436 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.276437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.277501 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.277504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.278062 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.278064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.278210 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.278764 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:53.278772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:53.280104 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:53.282135 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.282141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.282143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.282356 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.282576 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.282576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.282577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.282679 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.282982 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:53.282986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:53.283099 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.283180 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.283298 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260926 07:04:53.283473 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.274078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:40087 (local address 127.1.217.126:33381)
0926 07:04:53.274144 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.274146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.274377 (+ 231us) server_negotiation.cc:407] Connection header received
0926 07:04:53.275024 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.275025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.275041 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.275124 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.275750 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.276340 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.277595 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.277968 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.278045 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.280196 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.280197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.280197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.280205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.282047 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.282436 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.282438 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.282438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.282503 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.282748 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.282749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.282750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.282905 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.282995 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.283232 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.283337 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:04:53.284066 18099 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:53.284176 18099 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:53.284384 18099 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
I20260926 07:04:53.284899 17772 ts_manager.cc:194] Registered new tserver with Master: 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
I20260926 07:04:53.285279 17772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:40087
W20260926 07:04:53.357824 18102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:04:53.358016 18102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:04:53.358038 18102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:04:53.359191 18102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:04:53.359268 18102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:04:53.359315 18102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:04:53.360958 18102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:36139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=44439
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:04:53.361227 18102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:04:53.361470 18102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:04:53.366468 18107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:53.366465 18108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:04:53.366520 18110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:04:53.366662 18102 server_base.cc:1034] running on GCE node
I20260926 07:04:53.366923 18102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:04:53.367162 18102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:04:53.368333 18102 hybrid_clock.cc:648] HybridClock initialized: now 1790406293368312 us; error 31 us; skew 500 ppm
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:04:53.370817 18102 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:04:53.371254 18102 webserver.cc:460] Webserver started at http://127.1.217.67:44439/ using document root <none> and password file <none>
I20260926 07:04:53.371493 18102 fs_manager.cc:359] Metadata directory not provided
I20260926 07:04:53.371577 18102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:04:53.372939 18102 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:53.373584 18117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.373765 18102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:53.373864 18102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "40f33cdf1cb24d319b2d528378de6d16"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "921e946990c43043a397e68bf240189f"
server_key_iv: "a475d1a777514891e325a6153d32f708"
server_key_version: "encryptionkey@0"
I20260926 07:04:53.374236 18102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:04:53.391587 18102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:04:53.394162 18102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:04:53.394328 18102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:04:53.394569 18102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:04:53.394894 18102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:04:53.394953 18102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.395018 18102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:04:53.395048 18102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:04:53.401500 18102 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:36139
I20260926 07:04:53.401542 18230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:36139 every 8 connection(s)
I20260926 07:04:53.402067 18102 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
Sep 26 07:04:53 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406293, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:04:53.410020 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18102
I20260926 07:04:53.411769 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.402894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:48177 (local address 127.1.217.126:33381)
0926 07:04:53.402979 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.402982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.403217 (+ 235us) server_negotiation.cc:407] Connection header received
0926 07:04:53.403657 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.403658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.403667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.403723 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.404598 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.405154 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.406110 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.406300 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.406349 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.408383 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.408385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.408385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.408392 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.410281 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.410822 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.410824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.410824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.410893 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.411140 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.411141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.411142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.411302 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.411377 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.411643 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.411706 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 07:04:53.412011 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.402896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:48177)
0926 07:04:53.403078 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.403115 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.403573 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.403776 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.403780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.404330 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:04:53.404523 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.404527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.405237 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.405238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.406036 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.406038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.406372 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.406372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.406476 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.406953 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:04:53.406960 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:04:53.408278 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:04:53.410373 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.410380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.410382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.410732 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.410966 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:04:53.410967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:04:53.410967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:04:53.411068 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:04:53.411370 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:04:53.411375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:04:53.411489 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.411621 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.411786 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260926 07:04:53.412562 18231 heartbeater.cc:347] Connected to a master server at 127.1.217.126:33381
I20260926 07:04:53.412659 18231 heartbeater.cc:464] Registering TS with master...
I20260926 07:04:53.412884 18231 heartbeater.cc:511] Master 127.1.217.126:33381 requested a full tablet report, sending...
I20260926 07:04:53.413259 17772 ts_manager.cc:194] Registered new tserver with Master: 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139)
I20260926 07:04:53.413746 17772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:48177
I20260926 07:04:53.417215 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.410490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.1.217.126:33381)
0926 07:04:53.410611 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.410613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.410697 (+ 84us) server_negotiation.cc:407] Connection header received
0926 07:04:53.410729 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.410730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.410742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.410815 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.411573 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.412330 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.413389 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.413680 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.413749 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.415062 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.415064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.415064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.415070 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.416214 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.416530 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.416530 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.416531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.416579 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.416745 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.416746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.416746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.416911 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.417028 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.417077 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.417144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260926 07:04:53.418013 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:04:53.420838 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.418401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.1.217.126:33381)
0926 07:04:53.418439 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.418441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.418538 (+ 97us) server_negotiation.cc:407] Connection header received
0926 07:04:53.418595 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.418596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.418621 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.418679 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:53.418778 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.419213 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.419986 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.420175 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.420238 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.420356 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:53.420396 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 07:04:53.420734 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42446: token expired
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:04:53.420893 18235 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42446: token expired
W20260926 07:04:53.421293 17701 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.1.217.126:33381)
I20260926 07:04:53.428366 18235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.421533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.1.217.126:33381)
0926 07:04:53.421573 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.421575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.421583 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:04:53.421624 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.421625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.421635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.421727 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:04:53.422254 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.422690 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.423751 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.423922 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.423977 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.425579 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:04:53.425580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:04:53.425580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:04:53.425586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:04:53.427157 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.427645 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.427646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.427646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.427697 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:04:53.427907 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:04:53.427908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:04:53.427908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:04:53.428000 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:04:53.428072 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.428247 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.428305 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:04:53.429601 17772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:04:53.431175 17772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:04:53.433372 17894 tablet_service.cc:1467] Processing CreateTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 (DEFAULT_TABLE table=test-table [id=1912ae9bc6fa4402b033fd09952dc1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:53.433558 17894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065c14146c474bd5be0e49f9cdbeb745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:53.435097 17852 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Bootstrap starting.
I20260926 07:04:53.435438 17852 tablet_bootstrap.cc:654] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:53.436046 17852 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: No bootstrap required, opened a new log
I20260926 07:04:53.436085 17852 ts_tablet_manager.cc:1397] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:53.436456 17852 raft_consensus.cc:348] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.436539 17852 raft_consensus.cc:374] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:53.436583 17852 raft_consensus.cc:729] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bd22788b7b410fa1341267b58b8718, State: Initialized, Role: FOLLOWER
I20260926 07:04:53.436633 17852 consensus_queue.cc:260] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.436738 17852 ts_tablet_manager.cc:1428] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 07:04:53.437381 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.433146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:36139 (local address 127.0.0.1:60402)
0926 07:04:53.433346 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.433357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.433400 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.433852 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.433853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.433855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:53.433899 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.433901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.434962 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.434964 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.436165 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.436169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.437129 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.437130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.437199 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.437215 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.437267 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.437308 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 07:04:53.437685 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.433498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.1.217.67:36139)
0926 07:04:53.433658 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.433663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.433675 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:04:53.433714 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.433716 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.433732 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.433833 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:53.433980 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.434877 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.436264 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.437059 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.437133 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.437188 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.437550 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.437596 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260926 07:04:53.438293 18165 tablet_service.cc:1467] Processing CreateTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 (DEFAULT_TABLE table=test-table [id=1912ae9bc6fa4402b033fd09952dc1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:53.438616 18165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065c14146c474bd5be0e49f9cdbeb745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:53.438687 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.433200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.0.0.1:36602)
0926 07:04:53.433350 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.433360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.433400 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.435351 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.435352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.435353 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:53.435398 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.435399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.436560 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.436561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.437626 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.437628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.438448 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.438448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.438507 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.438512 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.438571 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.438610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
I20260926 07:04:53.438699 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.433323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.1.217.66:36263)
0926 07:04:53.435171 (+ 1848us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.435176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.435196 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:04:53.435240 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.435242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.435257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.435342 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:53.435458 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.436463 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.437719 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.438357 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.438447 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.438505 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.438585 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.438622 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":77,"threads_started":1}
I20260926 07:04:53.439251 18033 tablet_service.cc:1467] Processing CreateTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 (DEFAULT_TABLE table=test-table [id=1912ae9bc6fa4402b033fd09952dc1f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:04:53.439514 18033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065c14146c474bd5be0e49f9cdbeb745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:04:53.441254 17991 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Bootstrap starting.
I20260926 07:04:53.441745 18241 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Bootstrap starting.
I20260926 07:04:53.442090 17991 tablet_bootstrap.cc:654] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:53.442319 18241 tablet_bootstrap.cc:654] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Neither blocks nor log segments found. Creating new log.
I20260926 07:04:53.442324 17991 log.cc:824] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:53.442813 18241 log.cc:824] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Log is configured to *not* fsync() on all Append() calls
I20260926 07:04:53.442992 17991 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: No bootstrap required, opened a new log
I20260926 07:04:53.443053 17991 ts_tablet_manager.cc:1397] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:53.443542 18241 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: No bootstrap required, opened a new log
I20260926 07:04:53.443630 18241 ts_tablet_manager.cc:1397] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:04:53.444490 17991 raft_consensus.cc:348] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.444624 17991 raft_consensus.cc:374] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:53.444648 17991 raft_consensus.cc:729] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bf51ae30944ac1831277b3b7aee23f, State: Initialized, Role: FOLLOWER
I20260926 07:04:53.444794 17991 consensus_queue.cc:260] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.445101 17991 ts_tablet_manager.cc:1428] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:04:53.445133 18099 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
I20260926 07:04:53.445207 18241 raft_consensus.cc:348] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.445379 18241 raft_consensus.cc:374] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:04:53.445432 18241 raft_consensus.cc:729] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f33cdf1cb24d319b2d528378de6d16, State: Initialized, Role: FOLLOWER
I20260926 07:04:53.445572 18241 consensus_queue.cc:260] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.445863 18241 ts_tablet_manager.cc:1428] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:04:53.445874 18231 heartbeater.cc:503] Master 127.1.217.126:33381 was elected leader, sending a full tablet report...
W20260926 07:04:53.523988 18100 tablet.cc:2369] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:53.605427 18245 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:04:53.605590 18245 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.606012 18245 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263), b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:53.608911 17968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.606309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:40801 (local address 127.1.217.65:34389)
0926 07:04:53.606351 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.606352 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.606362 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:04:53.606424 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.606425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.606458 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.606537 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:53.606713 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.607202 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608131 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.608551 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608656 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.608736 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.608798 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.608840 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:04:53.608911 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.606221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.1.217.67:40801)
0926 07:04:53.606282 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.606295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.606346 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.606541 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.606542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.606544 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:53.606622 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.606624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.607303 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.607304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.608030 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.608032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.608643 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.608717 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.608720 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.608801 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.608841 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 07:04:53.609010 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.606313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:52371 (local address 127.1.217.66:36263)
0926 07:04:53.606361 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.606363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.606494 (+ 131us) server_negotiation.cc:407] Connection header received
0926 07:04:53.606565 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.606566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.606578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.606665 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:04:53.606833 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.607290 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608219 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.608581 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608672 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.608680 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.608877 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.608912 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:04:53.609074 18246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.606238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.1.217.67:52371)
0926 07:04:53.606421 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.606430 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:04:53.606508 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:04:53.606669 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:04:53.606669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:04:53.606671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:04:53.606761 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.606763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.607379 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.607380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.608118 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:04:53.608120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.608669 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.608670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:04:53.608753 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.608756 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:04:53.608870 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:04:53.608911 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260926 07:04:53.609274 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065c14146c474bd5be0e49f9cdbeb745" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bd22788b7b410fa1341267b58b8718" is_pre_election: true
I20260926 07:04:53.609406 17914 raft_consensus.cc:2454] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 0.
I20260926 07:04:53.609746 18053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065c14146c474bd5be0e49f9cdbeb745" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf51ae30944ac1831277b3b7aee23f" is_pre_election: true
I20260926 07:04:53.609792 18121 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f33cdf1cb24d319b2d528378de6d16, b9bd22788b7b410fa1341267b58b8718; no voters:
I20260926 07:04:53.609901 18053 raft_consensus.cc:2454] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 0.
I20260926 07:04:53.609929 18245 raft_consensus.cc:2781] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:04:53.609999 18245 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:04:53.610040 18245 raft_consensus.cc:3037] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:53.610771 18245 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.610930 18245 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 election: Requested vote from peers 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263), b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389)
I20260926 07:04:53.611078 18053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065c14146c474bd5be0e49f9cdbeb745" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75bf51ae30944ac1831277b3b7aee23f"
I20260926 07:04:53.611159 18053 raft_consensus.cc:3037] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:53.611109 17914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065c14146c474bd5be0e49f9cdbeb745" candidate_uuid: "40f33cdf1cb24d319b2d528378de6d16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9bd22788b7b410fa1341267b58b8718"
I20260926 07:04:53.611219 17914 raft_consensus.cc:3037] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:04:53.611779 18053 raft_consensus.cc:2454] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 1.
I20260926 07:04:53.611958 17914 raft_consensus.cc:2454] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40f33cdf1cb24d319b2d528378de6d16 in term 1.
I20260926 07:04:53.611955 18118 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f; no voters:
I20260926 07:04:53.612071 18245 raft_consensus.cc:2781] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:04:53.612231 18245 raft_consensus.cc:686] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 1 LEADER]: Becoming Leader. State: Replica: 40f33cdf1cb24d319b2d528378de6d16, State: Running, Role: LEADER
I20260926 07:04:53.612458 18245 consensus_queue.cc:237] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:04:53.613372 17772 catalog_manager.cc:5495] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67). New cstate: current_term: 1 leader_uuid: "40f33cdf1cb24d319b2d528378de6d16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } health_report { overall_health: UNKNOWN } } }
W20260926 07:04:53.624678 17961 tablet.cc:2369] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:04:53.652918 18232 tablet.cc:2369] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:04:53.663432 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.660936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.1.217.67:36139)
0926 07:04:53.661009 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 07:04:53.661011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:04:53.661024 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:04:53.661138 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:04:53.661139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:04:53.661177 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:04:53.661258 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:04:53.661405 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.661879 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.662615 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:04:53.662813 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:04:53.662926 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:04:53.663083 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:04:53.663197 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:04:53.663296 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 07:04:53.663316 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 07:04:53.663336 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 07:04:53.665232 18053 raft_consensus.cc:1261] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Refusing update from remote peer 40f33cdf1cb24d319b2d528378de6d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:53.665220 17914 raft_consensus.cc:1261] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Refusing update from remote peer 40f33cdf1cb24d319b2d528378de6d16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:04:53.665647 18247 consensus_queue.cc:1035] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:53.665797 18245 consensus_queue.cc:1035] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:04:53.668363 18251 mvcc.cc:205] Tried to move back new op lower bound from 7333504178849624064 to 7333504178637209600. Current Snapshot: MvccSnapshot[applied={T|T < 7333504178849624064 or (T in {7333504178849624064})}]
I20260926 07:04:53.668709 18250 mvcc.cc:205] Tried to move back new op lower bound from 7333504178849624064 to 7333504178637209600. Current Snapshot: MvccSnapshot[applied={T|T < 7333504178849624064}]
I20260926 07:04:53.670047 17771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:53.670152 17771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:04:53.670915 17771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Sending DeleteTablet for 3 replicas of tablet 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:04:53.671263 17894 tablet_service.cc:1514] Processing DeleteTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53706
I20260926 07:04:53.671294 18165 tablet_service.cc:1514] Processing DeleteTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:60402
I20260926 07:04:53.671309 18033 tablet_service.cc:1514] Processing DeleteTablet for tablet 065c14146c474bd5be0e49f9cdbeb745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:04:53 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:36602
I20260926 07:04:53.671516 18259 tablet_replica.cc:330] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: stopping tablet replica
I20260926 07:04:53.671515 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17810
I20260926 07:04:53.671617 18259 raft_consensus.cc:2229] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:04:53.671718 18259 raft_consensus.cc:2258] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:53.672144 18259 ts_tablet_manager.cc:1905] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:53.672888 18260 tablet_replica.cc:330] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: stopping tablet replica
I20260926 07:04:53.673036 18260 raft_consensus.cc:2229] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:04:53.673190 18260 raft_consensus.cc:2258] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:04:53.673645 18260 ts_tablet_manager.cc:1905] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:04:53.674192 18259 ts_tablet_manager.cc:1918] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:53.674256 18259 log.cc:1196] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/065c14146c474bd5be0e49f9cdbeb745
I20260926 07:04:53.674485 18259 ts_tablet_manager.cc:1939] T 065c14146c474bd5be0e49f9cdbeb745 P 75bf51ae30944ac1831277b3b7aee23f: Deleting consensus metadata
I20260926 07:04:53.675119 17756 catalog_manager.cc:4841] TS 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): tablet 065c14146c474bd5be0e49f9cdbeb745 (table test-table [id=1912ae9bc6fa4402b033fd09952dc1f3]) successfully deleted
I20260926 07:04:53.675598 18260 ts_tablet_manager.cc:1918] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:04:53.675673 18260 log.cc:1196] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/065c14146c474bd5be0e49f9cdbeb745
I20260926 07:04:53.675941 18260 ts_tablet_manager.cc:1939] T 065c14146c474bd5be0e49f9cdbeb745 P 40f33cdf1cb24d319b2d528378de6d16: Deleting consensus metadata
I20260926 07:04:53.676422 17757 catalog_manager.cc:4841] TS 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): tablet 065c14146c474bd5be0e49f9cdbeb745 (table test-table [id=1912ae9bc6fa4402b033fd09952dc1f3]) successfully deleted
W20260926 07:04:53.678362 17759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:34389 (error 108)
I20260926 07:04:53.678668 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17965
I20260926 07:04:53.678829 18237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:04:53.678709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34389 (local address 127.0.0.1:53720)
0926 07:04:53.678761 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 07:04:53.678795 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:04:53.681236 17759 catalog_manager.cc:4584] TS b9bd22788b7b410fa1341267b58b8718 (127.1.217.65:34389): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 065c14146c474bd5be0e49f9cdbeb745: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34389: connect: Connection refused (error 111)
I20260926 07:04:53.685869 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18102
I20260926 07:04:53.689924 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 17739
I20260926 07:05:13.695545 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--webserver_interface=127.1.217.126
--webserver_port=40985
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:33381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:13.775583 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:13.775801 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:13.775838 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:13.777292 18262 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 07:05:13.777333 18262 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 07:05:13.777347 18262 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 07:05:13.777359 18262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:13.777372 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:13.777385 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:13.777433 18262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:13.777462 18262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:13.778931 18262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:33381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:33381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=40985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:13.779177 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:13.779415 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:13.784199 18270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:13.784225 18267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:13.784207 18268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:13.784524 18262 server_base.cc:1034] running on GCE node
I20260926 07:05:13.784811 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:13.785111 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:13.786247 18262 hybrid_clock.cc:648] HybridClock initialized: now 1790406313786237 us; error 26 us; skew 500 ppm
Sep 26 07:05:13 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:13.788657 18262 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:13.789129 18262 webserver.cc:460] Webserver started at http://127.1.217.126:40985/ using document root <none> and password file <none>
I20260926 07:05:13.789356 18262 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:13.789422 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:13.790727 18262 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:13.791384 18277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:13.791582 18262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:13.791678 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "7f8061e5f4404117b4e355486a136e0c"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "e6e6281ba8ee154800589dc9c67f7e0c"
server_key_iv: "29497dc0a74c432f92f6f1c1f19d5be7"
server_key_version: "encryptionkey@0"
I20260926 07:05:13.792021 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:13.810228 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:13.812268 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:13.812425 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:13.816489 18262 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:33381
I20260926 07:05:13.816516 18329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:33381 every 8 connection(s)
I20260926 07:05:13.816977 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:13.818429 18330 sys_catalog.cc:263] Verifying existing consensus state
I20260926 07:05:13.819005 18330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap starting.
I20260926 07:05:13.820453 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18262
I20260926 07:05:13.820806 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:34389
--local_ip_for_outbound_sockets=127.1.217.65
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=45185
--webserver_interface=127.1.217.65
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:05:13.827651 18330 log.cc:824] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:13.830076 18330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:05:13.830410 18330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Bootstrap complete.
I20260926 07:05:13.832664 18330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:05:13.832957 18330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Initialized, Role: FOLLOWER
I20260926 07:05:13.833132 18330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:05:13.833187 18330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:13.833249 18330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:13.833302 18330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 2 FOLLOWER]: Advancing to term 3
I20260926 07:05:13.834343 18330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:05:13.834455 18330 leader_election.cc:302] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f8061e5f4404117b4e355486a136e0c; no voters:
I20260926 07:05:13.834620 18330 leader_election.cc:288] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 07:05:13.834688 18334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 07:05:13.834844 18334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [term 3 LEADER]: Becoming Leader. State: Replica: 7f8061e5f4404117b4e355486a136e0c, State: Running, Role: LEADER
I20260926 07:05:13.834892 18330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:13.834954 18334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } }
I20260926 07:05:13.835222 18334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f8061e5f4404117b4e355486a136e0c. Latest consensus state: current_term: 3 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:05:13.835417 18334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:13.835222 18335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7f8061e5f4404117b4e355486a136e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f8061e5f4404117b4e355486a136e0c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 33381 } } }
I20260926 07:05:13.835711 18335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:13.835714 18344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:13.836527 18344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:13.836627 18344 catalog_manager.cc:1259] Loaded cluster ID: 96f4f359d29f474da6be4bacb85b4a33
I20260926 07:05:13.836661 18344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:13.837929 18344 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:13.838116 18344 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Loaded TSK: 0
I20260926 07:05:13.838160 18344 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7f8061e5f4404117b4e355486a136e0c: Loaded TSK: 1
F20260926 07:05:13.838436 18344 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f503d98a9ad google::LogMessage::Fail() at ??:0
@ 0x7f503d98d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f503d98a4de google::LogMessage::Flush() at ??:0
@ 0x7f503d98ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f50400f7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f50400f4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f50458e41cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f50458e448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f50458e4594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f50458aac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f50458b4ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f503f8580d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f503f84d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f50406b16db start_thread at ??:0
@ 0x7f503c48861f clone at ??:0
W20260926 07:05:13.934904 18332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:13.935109 18332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:13.935132 18332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:13.936306 18332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:13.936348 18332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:13.936374 18332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:13.937852 18332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:34389
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=45185
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:13.938084 18332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:13.938318 18332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:13.944088 18371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:13.944178 18372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:13.944198 18374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:13.944460 18332 server_base.cc:1034] running on GCE node
I20260926 07:05:13.944675 18332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:13.944994 18332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:13.946174 18332 hybrid_clock.cc:648] HybridClock initialized: now 1790406313946155 us; error 30 us; skew 500 ppm
Sep 26 07:05:13 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:13.949052 18332 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:05:13.949584 18332 webserver.cc:460] Webserver started at http://127.1.217.65:45185/ using document root <none> and password file <none>
I20260926 07:05:13.949820 18332 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:13.949887 18332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:13.951517 18332 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 07:05:13.952236 18381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:13.952440 18332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:13.952531 18332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "b9bd22788b7b410fa1341267b58b8718"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "a7bde99fd74ec4c720ced85c5ae0bd7a"
server_key_iv: "fb291a4a929ed10b2d01f5f83bef0438"
server_key_version: "encryptionkey@0"
I20260926 07:05:13.952970 18332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:13.963996 18332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:13.966506 18332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:13.966650 18332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:13.966892 18332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:13.967398 18388 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 07:05:13.968815 18332 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 07:05:13.968873 18332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 07:05:13.968906 18332 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 07:05:13.969774 18332 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 07:05:13.969821 18332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 07:05:13.969894 18388 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Bootstrap starting.
I20260926 07:05:13.977250 18332 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:34389
I20260926 07:05:13.977409 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:34389 every 8 connection(s)
I20260926 07:05:13.977705 18332 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:05:13.980250 18388 log.cc:824] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:13.982870 18388 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 07:05:13.983184 18388 tablet_bootstrap.cc:492] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Bootstrap complete.
I20260926 07:05:13.984412 18388 ts_tablet_manager.cc:1397] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 07:05:13.985863 18388 raft_consensus.cc:348] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:13.986222 18388 raft_consensus.cc:729] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bd22788b7b410fa1341267b58b8718, State: Initialized, Role: FOLLOWER
I20260926 07:05:13.986246 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18332
I20260926 07:05:13.986370 18388 consensus_queue.cc:260] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:13.986553 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:36263
--local_ip_for_outbound_sockets=127.1.217.66
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=36633
--webserver_interface=127.1.217.66
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:05:13.986639 18388 ts_tablet_manager.cc:1428] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
W20260926 07:05:14.055110 18498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:05:13.978440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:55193)
0926 07:05:13.978695 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:13.978749 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:13.979427 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:14.054915 (+ 75488us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
W20260926 07:05:14.055364 18496 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:33381 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:33381: Client connection negotiation failed: client connection to 127.1.217.126:33381: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 07:05:14.055728 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.055601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:33801)
0926 07:05:14.055658 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.055706 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:05:14.056131 18498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.056027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:54681)
0926 07:05:14.056091 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.056114 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 07:05:14.056370 18496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 07:05:14.069310 18501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:14.069509 18501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:14.069533 18501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:14.070665 18501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:14.070708 18501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:14.070734 18501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:14.072139 18501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:36263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=36633
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:14.072374 18501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:14.072613 18501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:14.077641 18506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:14.077644 18507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:14.077666 18509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:14.077975 18501 server_base.cc:1034] running on GCE node
I20260926 07:05:14.078172 18501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:14.078403 18501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:14.079588 18501 hybrid_clock.cc:648] HybridClock initialized: now 1790406314079572 us; error 30 us; skew 500 ppm
Sep 26 07:05:14 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:14.082212 18501 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:05:14.082664 18501 webserver.cc:460] Webserver started at http://127.1.217.66:36633/ using document root <none> and password file <none>
I20260926 07:05:14.082854 18501 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:14.082897 18501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:14.084209 18501 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:14.084821 18516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.085040 18501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:14.085127 18501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "75bf51ae30944ac1831277b3b7aee23f"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "15ddf1637c930dc04786016d886768d7"
server_key_iv: "fcf3adbe84fae57d4e427df766305a6a"
server_key_version: "encryptionkey@0"
I20260926 07:05:14.085466 18501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:14.107438 18501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:14.109941 18501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:14.110059 18501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:14.110253 18501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:14.110581 18501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:14.110617 18501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.110641 18501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:14.110692 18501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.116366 18501 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:36263
I20260926 07:05:14.116458 18629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:36263 every 8 connection(s)
I20260926 07:05:14.116824 18501 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:05:14.117923 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.117527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:34801)
0926 07:05:14.117685 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.117759 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
W20260926 07:05:14.118330 18630 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:33381 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:33381: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
I20260926 07:05:14.118597 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.118512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:51603)
0926 07:05:14.118555 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.118580 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 07:05:14.118847 18632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.118778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:34429)
0926 07:05:14.118815 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.118834 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 07:05:14.118988 18630 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:05:14.121765 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18501
I20260926 07:05:14.122071 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:36139
--local_ip_for_outbound_sockets=127.1.217.67
--tserver_master_addrs=127.1.217.126:33381
--webserver_port=44439
--webserver_interface=127.1.217.67
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:32925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:14.205379 18633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:14.205591 18633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:14.205631 18633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:14.206745 18633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:14.206808 18633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:14.206853 18633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:14.208274 18633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:32925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:36139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=44439
--tserver_master_addrs=127.1.217.126:33381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:14.208541 18633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:14.208812 18633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:14.213653 18638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:14.213618 18641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:14.213614 18639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:14.213963 18633 server_base.cc:1034] running on GCE node
I20260926 07:05:14.214200 18633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:14.214434 18633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:14.215600 18633 hybrid_clock.cc:648] HybridClock initialized: now 1790406314215587 us; error 29 us; skew 500 ppm
Sep 26 07:05:14 dist-test-slave-t1mc krb5kdc[17192](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406314, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:14.218035 18633 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:05:14.218461 18633 webserver.cc:460] Webserver started at http://127.1.217.67:44439/ using document root <none> and password file <none>
I20260926 07:05:14.218695 18633 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:14.218765 18633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:14.220060 18633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:14.220646 18648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.220850 18633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:14.220955 18633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "40f33cdf1cb24d319b2d528378de6d16"
format_stamp: "Formatted at 2026-09-26 07:04:32 on dist-test-slave-t1mc"
server_key: "921e946990c43043a397e68bf240189f"
server_key_iv: "a475d1a777514891e325a6153d32f708"
server_key_version: "encryptionkey@0"
I20260926 07:05:14.221302 18633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:14.246043 18633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:14.247983 18633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:14.248152 18633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:14.248397 18633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:14.248710 18633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:14.248764 18633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.248824 18633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:14.248852 18633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:14.254585 18633 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:36139
I20260926 07:05:14.254637 18761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:36139 every 8 connection(s)
I20260926 07:05:14.254973 18633 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:14.255996 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.255592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:57581)
0926 07:05:14.255770 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.255853 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
W20260926 07:05:14.256448 18762 heartbeater.cc:644] Failed to heartbeat to 127.1.217.126:33381 (0 consecutive failures): Network error: Failed to ping master at 127.1.217.126:33381: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
I20260926 07:05:14.256832 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.256687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:50579)
0926 07:05:14.256751 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.256800 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:05:14.256875 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18633
I20260926 07:05:14.257228 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:14.257129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:56323)
0926 07:05:14.257178 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:14.257203 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:05:14.257421 18762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 07:05:15.057192 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.056941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:44151)
0926 07:05:15.057092 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:15.057131 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":68,"threads_started":1}
I20260926 07:05:15.119479 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.119275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:50059)
0926 07:05:15.119394 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:15.119424 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260926 07:05:15.155565 18768 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:15.155686 18768 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:15.156080 18768 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139), 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
Sep 26 07:05:15 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.67@KRBTEST.COM
Sep 26 07:05:15 dist-test-slave-t1mc krb5kdc[17192](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406313, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.66@KRBTEST.COM
I20260926 07:05:15.166618 18769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.156267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36263 (local address 127.1.217.65:42345)
0926 07:05:15.156443 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:15.156454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:15.156534 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:15.157348 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:15.157349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:15.157764 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:15.157966 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:15.157971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.159042 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.159044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:15.159828 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:15.159830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.160594 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.160595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:15.160643 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:15.161265 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:15.161273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:15.163060 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:15.165076 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:15.165077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:15.165303 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:15.165532 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:15.165533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:15.165607 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:15.166003 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:15.166010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:15.166161 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 07:05:15.166287 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 07:05:15.166465 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":133,"thread_start_us":70,"threads_started":1}
I20260926 07:05:15.166708 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.156260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:36139 (local address 127.1.217.65:45665)
0926 07:05:15.156322 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:15.156335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:15.156411 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:15.157285 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:15.157288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:15.157763 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:15.157966 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:15.157971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.159075 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.159076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:15.159844 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:15.159846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.160515 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.160516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:15.160634 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:15.161326 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:15.161331 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:15.162712 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:15.164866 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:15.164872 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:15.164873 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:15.165168 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:15.165415 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:15.165416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:15.165543 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:15.166003 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:15.166015 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:15.166161 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 07:05:15.166287 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 07:05:15.166548 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":111}
I20260926 07:05:15.166666 18770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.156379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:42345 (local address 127.1.217.66:36263)
0926 07:05:15.156604 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:15.156614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:15.156646 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:05:15.157141 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:15.157143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:15.157201 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:15.157337 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:15.158095 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.158938 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.159916 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.160427 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.160557 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:15.163180 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:15.163182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:15.163183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:15.163192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:15.165009 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165374 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:15.165377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:15.165377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:15.165470 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165673 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:15.165674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:15.165674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:15.165928 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:15.166014 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:15.166390 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 07:05:15.166536 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":64,"threads_started":1}
I20260926 07:05:15.167244 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.156415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:45665 (local address 127.1.217.67:36139)
0926 07:05:15.156572 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:15.156587 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:15.156607 (+ 20us) server_negotiation.cc:407] Connection header received
0926 07:05:15.157053 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:15.157055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:15.157110 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:15.157218 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:15.158085 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.158950 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.159942 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:15.160426 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:15.160513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:15.162820 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:15.162824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:15.162825 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:15.162843 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:15.164738 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165243 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:15.165246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:15.165246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:15.165347 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:15.165611 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:15.165612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:15.165612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:15.165928 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:15.166014 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:15.166464 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 07:05:15.166592 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
W20260926 07:05:15.167521 18382 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
W20260926 07:05:15.167917 18383 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:05:15.167971 18383 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9bd22788b7b410fa1341267b58b8718; no voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f
I20260926 07:05:15.168090 18768 raft_consensus.cc:2726] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:05:15.257946 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:15.257717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:40829)
0926 07:05:15.257834 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:15.257903 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260926 07:05:16.057940 18773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:16.057715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:56461)
0926 07:05:16.057853 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:16.057892 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260926 07:05:16.120268 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:16.120054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:56511)
0926 07:05:16.120190 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:16.120228 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260926 07:05:16.258585 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:16.258393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:55019)
0926 07:05:16.258501 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:16.258547 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260926 07:05:16.979710 18776 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:16.979790 18776 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:16.979933 18776 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139), 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
W20260926 07:05:16.980322 18383 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
W20260926 07:05:16.980403 18382 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:05:16.980466 18382 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9bd22788b7b410fa1341267b58b8718; no voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f
I20260926 07:05:16.980541 18776 raft_consensus.cc:2726] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:05:17.058665 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:17.058417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:43447)
0926 07:05:17.058558 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:17.058595 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":75,"threads_started":1}
I20260926 07:05:17.121045 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:17.120862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:46321)
0926 07:05:17.120975 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:17.121006 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260926 07:05:17.259203 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:17.259003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:36443)
0926 07:05:17.259127 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:17.259161 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":70,"threads_started":1}
I20260926 07:05:18.059396 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:18.059150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:46875)
0926 07:05:18.059287 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:18.059346 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260926 07:05:18.121702 18781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:18.121506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:60565)
0926 07:05:18.121628 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:18.121664 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 07:05:18.260022 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:18.259780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:44963)
0926 07:05:18.259920 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:18.259950 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260926 07:05:19.060151 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:19.059945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:36405)
0926 07:05:19.060070 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:19.060106 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260926 07:05:19.122296 18784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:19.122112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:44681)
0926 07:05:19.122227 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:19.122260 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260926 07:05:19.260694 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:19.260477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:57899)
0926 07:05:19.260611 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:19.260651 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260926 07:05:19.972582 18786 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:19.972675 18786 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:19.972903 18786 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139), 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
W20260926 07:05:19.973284 18383 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
W20260926 07:05:19.973364 18382 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:05:19.973416 18382 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9bd22788b7b410fa1341267b58b8718; no voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f
I20260926 07:05:19.973491 18786 raft_consensus.cc:2726] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:05:20.060937 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:20.060650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:36875)
0926 07:05:20.060835 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:20.060890 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":73,"threads_started":1}
I20260926 07:05:20.123080 18788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:20.122819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:49135)
0926 07:05:20.122994 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:20.123035 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260926 07:05:20.261353 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:20.261164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:59287)
0926 07:05:20.261277 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:20.261314 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 07:05:21.061640 18790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:21.061398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:47693)
0926 07:05:21.061556 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:21.061594 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 07:05:21.123759 18791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:21.123572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:39085)
0926 07:05:21.123689 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:21.123719 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 07:05:21.261966 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:21.261761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:54957)
0926 07:05:21.261881 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:21.261926 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260926 07:05:22.062307 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:22.062102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:39233)
0926 07:05:22.062224 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:22.062260 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260926 07:05:22.124595 18794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:22.124382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:44161)
0926 07:05:22.124511 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:22.124551 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260926 07:05:22.262589 18795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:22.262355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:36123)
0926 07:05:22.262516 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:22.262551 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260926 07:05:22.736212 18796 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:22.736325 18796 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:22.736492 18796 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139), 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
W20260926 07:05:22.736865 18383 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
W20260926 07:05:22.736931 18382 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:05:22.737018 18382 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9bd22788b7b410fa1341267b58b8718; no voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f
I20260926 07:05:22.737093 18796 raft_consensus.cc:2726] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:05:23.062976 18797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:23.062758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:45349)
0926 07:05:23.062895 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:23.062929 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":60,"threads_started":1}
I20260926 07:05:23.125250 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:23.125029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:37591)
0926 07:05:23.125159 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:23.125194 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260926 07:05:23.263249 18799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:23.263029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:40517)
0926 07:05:23.263168 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:23.263207 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260926 07:05:24.063629 18800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:24.063385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:59871)
0926 07:05:24.063543 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:24.063584 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":63,"threads_started":1}
I20260926 07:05:24.125874 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:24.125675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:39287)
0926 07:05:24.125800 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:24.125828 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":71,"threads_started":1}
I20260926 07:05:24.263952 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:24.263728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:33129)
0926 07:05:24.263865 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:24.263901 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260926 07:05:24.633752 18803 raft_consensus.cc:482] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:24.633843 18803 raft_consensus.cc:504] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40f33cdf1cb24d319b2d528378de6d16" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 36139 } } peers { permanent_uuid: "75bf51ae30944ac1831277b3b7aee23f" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36263 } } peers { permanent_uuid: "b9bd22788b7b410fa1341267b58b8718" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34389 } }
I20260926 07:05:24.634034 18803 leader_election.cc:288] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139), 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263)
W20260926 07:05:24.634447 18383 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 40f33cdf1cb24d319b2d528378de6d16 (127.1.217.67:36139): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
W20260926 07:05:24.634506 18382 leader_election.cc:341] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 75bf51ae30944ac1831277b3b7aee23f (127.1.217.66:36263): Not found: Tablet not found: 065c14146c474bd5be0e49f9cdbeb745
I20260926 07:05:24.634541 18382 leader_election.cc:302] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9bd22788b7b410fa1341267b58b8718; no voters: 40f33cdf1cb24d319b2d528378de6d16, 75bf51ae30944ac1831277b3b7aee23f
I20260926 07:05:24.634635 18803 raft_consensus.cc:2726] T 065c14146c474bd5be0e49f9cdbeb745 P b9bd22788b7b410fa1341267b58b8718 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 07:05:25.064318 18804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:25.064112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:51275)
0926 07:05:25.064236 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:25.064274 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260926 07:05:25.126538 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:25.126332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:33499)
0926 07:05:25.126458 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:25.126496 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 07:05:25.264696 18806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:25.264468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:45519)
0926 07:05:25.264619 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:25.264655 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
I20260926 07:05:26.065116 18807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:26.064915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:35963)
0926 07:05:26.065037 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:26.065070 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260926 07:05:26.127183 18808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:26.126978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:57241)
0926 07:05:26.127108 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:26.127140 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260926 07:05:26.265347 18809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:26.265164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:48357)
0926 07:05:26.265269 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:26.265303 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":57,"threads_started":1}
I20260926 07:05:27.065750 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:27.065555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:60065)
0926 07:05:27.065668 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:27.065698 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 07:05:27.127791 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:27.127596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:35561)
0926 07:05:27.127713 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:27.127749 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260926 07:05:27.265939 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:27.265729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:38145)
0926 07:05:27.265859 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:27.265894 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260926 07:05:28.066455 18813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:28.066219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:50747)
0926 07:05:28.066370 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:28.066409 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":68,"threads_started":1}
I20260926 07:05:28.128435 18814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:28.128234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:35149)
0926 07:05:28.128349 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:28.128382 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260926 07:05:28.266625 18815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:28.266356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:40007)
0926 07:05:28.266548 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:28.266585 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":56,"threads_started":1}
I20260926 07:05:29.067240 18816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:29.067040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.65:41547)
0926 07:05:29.067156 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:29.067195 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260926 07:05:29.129117 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:29.128935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.66:44755)
0926 07:05:29.129045 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:29.129075 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260926 07:05:29.261291 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18332
I20260926 07:05:29.266103 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18501
I20260926 07:05:29.267685 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:29.267475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:33381 (local address 127.1.217.67:34123)
0926 07:05:29.267594 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:29.267631 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.126:33381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260926 07:05:29.270391 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18633
2026-09-26T07:05:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58451 ms)
[----------] 30 tests from SecurityITest (227369 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:29 dist-test-slave-t1mc krb5kdc[18821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:29 dist-test-slave-t1mc krb5kdc[18821](info): set up 2 sockets
Sep 26 07:05:29 dist-test-slave-t1mc krb5kdc[18821](info): commencing operation
krb5kdc: starting...
W20260926 07:05:30.312485 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:30.431819 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46549
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:30.512557 18834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:30.512825 18834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:30.512866 18834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:30.514009 18834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:30.514046 18834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:30.514060 18834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:30.514075 18834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:30.514087 18834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:30.515501 18834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:46549
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:30.515736 18834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:30.515947 18834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:05:30.521166 18834 server_base.cc:1034] running on GCE node
W20260926 07:05:30.521301 18842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:30.521337 18840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:30.521351 18839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:30.521782 18834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:30.522113 18834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:30.523283 18834 hybrid_clock.cc:648] HybridClock initialized: now 1790406330523252 us; error 51 us; skew 500 ppm
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:30.525954 18834 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:30.526399 18834 webserver.cc:460] Webserver started at http://127.1.217.126:41737/ using document root <none> and password file <none>
I20260926 07:05:30.526634 18834 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:30.526700 18834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:30.526850 18834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:30.527915 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "c9c7e337f15d475e840db22c2f14e01b"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "ad4fb6d1c82bf73512bf911c6cb46265"
server_key_iv: "0978e0c2e38d0a4bcfda0cd7eae30609"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.528265 18834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "c9c7e337f15d475e840db22c2f14e01b"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "ad4fb6d1c82bf73512bf911c6cb46265"
server_key_iv: "0978e0c2e38d0a4bcfda0cd7eae30609"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.529727 18834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:30.530591 18849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.530799 18834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:30.530913 18834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "c9c7e337f15d475e840db22c2f14e01b"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "ad4fb6d1c82bf73512bf911c6cb46265"
server_key_iv: "0978e0c2e38d0a4bcfda0cd7eae30609"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.530987 18834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:30.545193 18834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:30.547266 18834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:30.547429 18834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:30.551501 18834 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:46549
I20260926 07:05:30.551535 18901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:46549 every 8 connection(s)
I20260926 07:05:30.551920 18834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:30.552903 18902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:30.555137 18902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: Bootstrap starting.
I20260926 07:05:30.555658 18902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:30.555896 18902 log.cc:824] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:30.556479 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18834
I20260926 07:05:30.556574 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:30.556586 18902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: No bootstrap required, opened a new log
I20260926 07:05:30.556806 1893 external_mini_cluster.cc:1428] Setting key 87659cfbe201dd1f3895bb36469e484f
I20260926 07:05:30.559056 18902 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } }
I20260926 07:05:30.559204 18902 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:30.559252 18902 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c7e337f15d475e840db22c2f14e01b, State: Initialized, Role: FOLLOWER
I20260926 07:05:30.559391 18902 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } }
I20260926 07:05:30.559468 18902 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:30.559511 18902 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:30.559558 18902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:30.560081 18902 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } }
I20260926 07:05:30.560194 18902 leader_election.cc:302] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9c7e337f15d475e840db22c2f14e01b; no voters:
I20260926 07:05:30.560377 18902 leader_election.cc:288] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:30.560411 18907 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:30.560583 18907 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [term 1 LEADER]: Becoming Leader. State: Replica: c9c7e337f15d475e840db22c2f14e01b, State: Running, Role: LEADER
I20260926 07:05:30.560631 18902 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:30.560704 18907 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } }
I20260926 07:05:30.563562 18909 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9c7e337f15d475e840db22c2f14e01b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } } }
I20260926 07:05:30.563683 18909 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:30.563640 18908 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9c7e337f15d475e840db22c2f14e01b. Latest consensus state: current_term: 1 leader_uuid: "c9c7e337f15d475e840db22c2f14e01b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9c7e337f15d475e840db22c2f14e01b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46549 } } }
I20260926 07:05:30.563757 18908 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b [sys.catalog]: This master's current role is: LEADER
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
W20260926 07:05:30.564201 18921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:30.564265 18921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:30.564534 18922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:30.564728 18922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:30.566229 18922 catalog_manager.cc:1347] Generated new cluster ID: 4f0b2d8d288d430b8cf6dae4cac0ecb1
I20260926 07:05:30.566282 18922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:30.567997 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:30.557314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39496 (local address 127.1.217.126:46549)
0926 07:05:30.557477 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:30.557492 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:30.557524 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:05:30.557969 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:30.557975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:30.558021 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:30.558133 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:30.558907 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.560166 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.561317 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.561808 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.561885 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:30.564577 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:30.564581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:30.564581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:30.564603 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:30.566532 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.566883 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.566886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.566887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.566973 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.567172 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.567173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.567173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.567404 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:30.567571 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:30.567642 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 07:05:30.567773 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 07:05:30.575912 18922 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:30.576437 18922 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:30.589639 18922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: Generated new TSK 0
I20260926 07:05:30.589862 18922 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:05:30.624159 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:30.721243 18928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:30.721441 18928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:30.721465 18928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:30.722604 18928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:30.722643 18928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:30.722671 18928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:30.724349 18928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:30.724525 18928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:30.724690 18928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:30.729583 18936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:30.729732 18934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:30.729774 18933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:30.729743 18928 server_base.cc:1034] running on GCE node
I20260926 07:05:30.730068 18928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:30.730301 18928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:30.731455 18928 hybrid_clock.cc:648] HybridClock initialized: now 1790406330731436 us; error 38 us; skew 500 ppm
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:30.733958 18928 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:05:30.734429 18928 webserver.cc:460] Webserver started at http://127.1.217.65:34377/ using document root <none> and password file <none>
I20260926 07:05:30.734655 18928 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:30.734717 18928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:30.734848 18928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:30.735846 18928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "aa32276c1a264f1c8b76981b873e8384"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "775e6f0fbc4626a97d5b4aaeee34755a"
server_key_iv: "645260879366b1209fbf1cedaf7b9b49"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.736210 18928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "aa32276c1a264f1c8b76981b873e8384"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "775e6f0fbc4626a97d5b4aaeee34755a"
server_key_iv: "645260879366b1209fbf1cedaf7b9b49"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.737586 18928 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:30.738301 18943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.738461 18928 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:30.738554 18928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "aa32276c1a264f1c8b76981b873e8384"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "775e6f0fbc4626a97d5b4aaeee34755a"
server_key_iv: "645260879366b1209fbf1cedaf7b9b49"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.738633 18928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:30.755180 18928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:30.757226 18928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:30.757380 18928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:30.757609 18928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:30.757908 18928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:30.757957 18928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.758009 18928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:30.758038 18928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.764256 18928 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:34301
I20260926 07:05:30.764302 19056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:34301 every 8 connection(s)
I20260926 07:05:30.764616 18928 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:05:30.769765 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 18928
I20260926 07:05:30.769878 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:30.770077 1893 external_mini_cluster.cc:1428] Setting key 5d744525966c0c8357716084c41e5f70
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:30.774554 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:30.765335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:45135 (local address 127.1.217.126:46549)
0926 07:05:30.765399 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:30.765401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:30.765585 (+ 184us) server_negotiation.cc:407] Connection header received
0926 07:05:30.766118 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:30.766119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:30.766130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:30.766193 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:30.767107 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.767614 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.768597 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.768920 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.768984 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:30.771595 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:30.771597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:30.771597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:30.771604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:30.773168 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.773590 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.773592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.773592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.773651 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.773897 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.773898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.773899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.774021 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:30.774090 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:30.774369 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 07:05:30.774454 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:05:30.774768 19059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:30.765306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46549 (local address 127.1.217.65:45135)
0926 07:05:30.765467 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:30.765502 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:30.766008 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:30.766248 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:30.766252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:30.766805 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:30.767004 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:30.767008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.767695 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.767696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:30.768509 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:30.768511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.769023 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.769024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:30.769148 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:30.769651 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:30.769662 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:30.771491 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:30.773263 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:30.773271 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:30.773273 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:30.773504 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:30.773715 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:30.773716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:30.773716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:30.773833 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:30.774092 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:30.774099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:30.774240 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 07:05:30.774348 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 07:05:30.774523 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260926 07:05:30.775254 19057 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46549
I20260926 07:05:30.775381 19057 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:30.775617 19057 heartbeater.cc:511] Master 127.1.217.126:46549 requested a full tablet report, sending...
I20260926 07:05:30.776165 18866 ts_manager.cc:194] Registered new tserver with Master: aa32276c1a264f1c8b76981b873e8384 (127.1.217.65:34301)
I20260926 07:05:30.776824 18866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:45135
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:05:30.797390 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:30.879436 19062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:30.879709 19062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:30.879750 19062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:30.880920 19062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:30.880985 19062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:30.881031 19062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:30.882535 19062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:30.882805 19062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:30.883035 19062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:30.888130 19070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:30.888114 19067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:30.888155 19062 server_base.cc:1034] running on GCE node
W20260926 07:05:30.888103 19068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:30.888587 19062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:30.888881 19062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:30.890038 19062 hybrid_clock.cc:648] HybridClock initialized: now 1790406330890017 us; error 35 us; skew 500 ppm
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:30.892493 19062 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:05:30.892999 19062 webserver.cc:460] Webserver started at http://127.1.217.66:35563/ using document root <none> and password file <none>
I20260926 07:05:30.893229 19062 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:30.893272 19062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:30.893415 19062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:30.894443 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "b54632ca938b893581c084f9014af489"
server_key_iv: "06bde78127a07f07546f0d85574f3376"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.894799 19062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "b54632ca938b893581c084f9014af489"
server_key_iv: "06bde78127a07f07546f0d85574f3376"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.896107 19062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:30.896883 19077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.897054 19062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:30.897150 19062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d"
format_stamp: "Formatted at 2026-09-26 07:05:30 on dist-test-slave-t1mc"
server_key: "b54632ca938b893581c084f9014af489"
server_key_iv: "06bde78127a07f07546f0d85574f3376"
server_key_version: "encryptionkey@0"
I20260926 07:05:30.897230 19062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:30.905786 19062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:30.907536 19062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:30.907685 19062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:30.907892 19062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:30.908176 19062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:30.908226 19062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.908267 19062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:30.908294 19062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:30.914222 19062 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:46631
I20260926 07:05:30.914271 19190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:46631 every 8 connection(s)
I20260926 07:05:30.914564 19062 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:05:30 dist-test-slave-t1mc krb5kdc[18821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406330, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:30.921998 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19062
I20260926 07:05:30.922106 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:30.922328 1893 external_mini_cluster.cc:1428] Setting key 9f6c18e0b9a1a31fabeaaed32b60dea3
I20260926 07:05:30.924225 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:30.915247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:41851 (local address 127.1.217.126:46549)
0926 07:05:30.915347 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:30.915350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:30.915549 (+ 199us) server_negotiation.cc:407] Connection header received
0926 07:05:30.916079 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:30.916081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:30.916093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:30.916149 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:30.917170 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.917670 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.918665 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.918863 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.918952 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:30.921002 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:30.921003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:30.921004 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:30.921012 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:30.922519 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.923092 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.923093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.923093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.923164 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:30.923429 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:30.923430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:30.923431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:30.923581 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:30.923659 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:30.923992 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 07:05:30.924083 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 07:05:30.924248 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:30.915219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46549 (local address 127.1.217.66:41851)
0926 07:05:30.915432 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:30.915467 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:30.915975 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:30.916217 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:30.916221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:30.916869 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:30.917068 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:30.917073 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.917761 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.917762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:30.918560 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:30.918562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:30.918961 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:30.918962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:30.919086 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:30.919546 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:30.919555 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:30.920908 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:30.922621 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:30.922628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:30.922630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:30.922898 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:30.923238 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:30.923239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:30.923239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:30.923362 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:30.923653 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:30.923658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:30.923840 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 07:05:30.923949 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 07:05:30.924089 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260926 07:05:30.924815 19191 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46549
I20260926 07:05:30.924913 19191 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:30.925084 19191 heartbeater.cc:511] Master 127.1.217.126:46549 requested a full tablet report, sending...
I20260926 07:05:30.925488 18866 ts_manager.cc:194] Registered new tserver with Master: 3d6fbb68aa4b4364ac59d2aff2bec43d (127.1.217.66:46631)
I20260926 07:05:30.925860 18866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:41851
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:05:30.947753 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:31.028190 19196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:31.028396 19196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:31.028420 19196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:31.029585 19196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:31.029636 19196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:31.029662 19196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:31.031073 19196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:46549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:31.031327 19196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:31.031558 19196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:31.036595 19204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.036576 19202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.036566 19201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.036720 19196 server_base.cc:1034] running on GCE node
I20260926 07:05:31.037053 19196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:31.037320 19196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:31.038484 19196 hybrid_clock.cc:648] HybridClock initialized: now 1790406331038467 us; error 26 us; skew 500 ppm
Sep 26 07:05:31 dist-test-slave-t1mc krb5kdc[18821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:31.041059 19196 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:05:31.041631 19196 webserver.cc:460] Webserver started at http://127.1.217.67:39935/ using document root <none> and password file <none>
I20260926 07:05:31.041894 19196 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:31.041970 19196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:31.042119 19196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:31.043218 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "4b0755ec8a7b40e0aa19a6057f4918f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "391f5359a7ad4b7cdc90c29aff503848"
server_key_iv: "7b5347f327aee803b009e47f2c615abd"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.043583 19196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "4b0755ec8a7b40e0aa19a6057f4918f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "391f5359a7ad4b7cdc90c29aff503848"
server_key_iv: "7b5347f327aee803b009e47f2c615abd"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.045006 19196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:31.045830 19211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.046039 19196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:31.046136 19196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "4b0755ec8a7b40e0aa19a6057f4918f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "391f5359a7ad4b7cdc90c29aff503848"
server_key_iv: "7b5347f327aee803b009e47f2c615abd"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.046216 19196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:31.061506 19196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:31.063678 19196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:31.063858 19196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:31.064234 19196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:31.064556 19196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:31.064611 19196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.064654 19196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:31.064680 19196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.070796 19196 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:46339
I20260926 07:05:31.070838 19324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:46339 every 8 connection(s)
I20260926 07:05:31.071162 19196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:31.072566 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19196
I20260926 07:05:31.072696 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:31.072940 1893 external_mini_cluster.cc:1428] Setting key 133579738d876156f6bae8b0d57a1262
Sep 26 07:05:31 dist-test-slave-t1mc krb5kdc[18821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406331, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:31.081213 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.071885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:47539 (local address 127.1.217.126:46549)
0926 07:05:31.071951 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.071953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.072174 (+ 221us) server_negotiation.cc:407] Connection header received
0926 07:05:31.072669 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.072671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.072687 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.072763 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.074119 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.074628 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.075630 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.075812 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.075866 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.078176 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.078178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.078178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:31.078186 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.079658 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:31.080093 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:31.080094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:31.080094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:31.080147 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:31.080452 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:31.080453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:31.080454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:31.080571 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.080628 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.081054 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.081142 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:05:31.081369 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.071883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:46549 (local address 127.1.217.67:47539)
0926 07:05:31.072041 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.072075 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.072551 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.072869 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.072873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.073737 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:31.074023 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.074031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.074709 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.074709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.075558 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.075560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.075910 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.075911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.076022 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.076531 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:31.076539 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:31.078081 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:31.079748 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:31.079756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:31.079759 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:31.080026 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:31.080224 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:31.080225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:31.080226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:31.080375 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:31.080653 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:31.080660 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:31.080872 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.081002 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.081144 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260926 07:05:31.081868 19325 heartbeater.cc:347] Connected to a master server at 127.1.217.126:46549
I20260926 07:05:31.081966 19325 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:31.082123 19325 heartbeater.cc:511] Master 127.1.217.126:46549 requested a full tablet report, sending...
I20260926 07:05:31.082505 18866 ts_manager.cc:194] Registered new tserver with Master: 4b0755ec8a7b40e0aa19a6057f4918f2 (127.1.217.67:46339)
I20260926 07:05:31.082847 18866 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:47539
I20260926 07:05:31.084896 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:31.092213 18906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.086222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39500 (local address 127.1.217.126:46549)
0926 07:05:31.086281 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.086284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.086396 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:05:31.086475 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.086476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.086485 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.086549 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.087141 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.087621 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.088348 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.088543 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.088616 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.089796 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.089798 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.089798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:31.089805 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.091063 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:31.091407 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:31.091408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:31.091408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:31.091472 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:31.091664 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:31.091665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:31.091666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:31.091817 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.091933 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.092026 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.092108 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:05:31.093900 18866 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:31.095513 18866 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:31.102475 19336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46631 (local address 127.0.0.1:58230)
0926 07:05:31.098216 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.098254 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.098317 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.098672 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.098675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.098678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.098750 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.098753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.100174 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.100175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.101523 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.101525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102217 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.102279 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102281 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.102329 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.102392 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":44,"thread_start_us":58,"threads_started":1}
I20260926 07:05:31.102536 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.1.217.67:46339)
0926 07:05:31.098487 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.098490 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.098666 (+ 176us) server_negotiation.cc:407] Connection header received
0926 07:05:31.098790 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.098793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.098811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.098863 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.099119 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.100199 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.101367 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102086 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102163 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102225 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.102407 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.102450 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260926 07:05:31.102695 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.1.217.65:34301)
0926 07:05:31.098600 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.098604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.098613 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:31.098650 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.098652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.098666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.098738 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.098931 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.099841 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.101467 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102227 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102292 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102344 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.102444 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.102484 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":126,"threads_started":1}
I20260926 07:05:31.102762 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:46339 (local address 127.0.0.1:48390)
0926 07:05:31.098607 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.098616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.098694 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.099001 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.099002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.099004 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.099044 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.099045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.100307 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.100308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.101260 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.101262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102179 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.102261 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102279 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.102657 (+ 378us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.102700 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":7,"thread_start_us":79,"threads_started":1}
I20260926 07:05:31.102850 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.1.217.66:46631)
0926 07:05:31.098484 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.098490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.098500 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:31.098541 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.098544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.098562 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.098830 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.098923 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.100077 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.101593 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102105 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102612 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102688 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.102714 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.102759 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":72,"threads_started":1}
I20260926 07:05:31.103413 19259 tablet_service.cc:1467] Processing CreateTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 (DEFAULT_TABLE table=test-table [id=406f6ca3bcc94a638ba2168374e0b3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.103472 19125 tablet_service.cc:1467] Processing CreateTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 (DEFAULT_TABLE table=test-table [id=406f6ca3bcc94a638ba2168374e0b3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.103724 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ca9d825b9e4b0b8b446557b2b38109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.103731 19259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ca9d825b9e4b0b8b446557b2b38109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.105103 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.098057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34301 (local address 127.0.0.1:35834)
0926 07:05:31.098463 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.098475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.098528 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.098815 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.098815 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.098817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.098850 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.098851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.099937 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.099939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.101370 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.101373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.102293 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.102294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.102366 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.102371 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.102428 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.102471 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":9,"thread_start_us":97,"threads_started":1}
I20260926 07:05:31.105803 18991 tablet_service.cc:1467] Processing CreateTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 (DEFAULT_TABLE table=test-table [id=406f6ca3bcc94a638ba2168374e0b3cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.106132 18991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ca9d825b9e4b0b8b446557b2b38109. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.106549 19342 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Bootstrap starting.
I20260926 07:05:31.106701 19343 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Bootstrap starting.
I20260926 07:05:31.107167 19342 tablet_bootstrap.cc:654] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.107290 19343 tablet_bootstrap.cc:654] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.107565 19342 log.cc:824] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.107875 19343 log.cc:824] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.108321 19342 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: No bootstrap required, opened a new log
I20260926 07:05:31.108369 19342 ts_tablet_manager.cc:1397] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:31.109411 19343 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: No bootstrap required, opened a new log
I20260926 07:05:31.109483 19343 ts_tablet_manager.cc:1397] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 07:05:31.110275 19342 raft_consensus.cc:348] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.110431 19342 raft_consensus.cc:374] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.110488 19342 raft_consensus.cc:729] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6fbb68aa4b4364ac59d2aff2bec43d, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.110606 19342 consensus_queue.cc:260] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.110927 19342 ts_tablet_manager.cc:1428] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 07:05:31.110940 19191 heartbeater.cc:503] Master 127.1.217.126:46549 was elected leader, sending a full tablet report...
I20260926 07:05:31.110931 19343 raft_consensus.cc:348] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.111109 19343 raft_consensus.cc:374] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.111148 19343 raft_consensus.cc:729] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b0755ec8a7b40e0aa19a6057f4918f2, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.111150 19346 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: Bootstrap starting.
I20260926 07:05:31.111295 19343 consensus_queue.cc:260] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.111593 19325 heartbeater.cc:503] Master 127.1.217.126:46549 was elected leader, sending a full tablet report...
I20260926 07:05:31.111596 19343 ts_tablet_manager.cc:1428] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:31.111805 19346 tablet_bootstrap.cc:654] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.112421 19346 log.cc:824] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.113125 19346 tablet_bootstrap.cc:492] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: No bootstrap required, opened a new log
I20260926 07:05:31.113179 19346 ts_tablet_manager.cc:1397] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:31.114607 19346 raft_consensus.cc:348] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.114740 19346 raft_consensus.cc:374] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.114763 19346 raft_consensus.cc:729] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa32276c1a264f1c8b76981b873e8384, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.114892 19346 consensus_queue.cc:260] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.115121 19346 ts_tablet_manager.cc:1428] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:31.115190 19057 heartbeater.cc:503] Master 127.1.217.126:46549 was elected leader, sending a full tablet report...
I20260926 07:05:31.159821 19348 raft_consensus.cc:482] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:31.159966 19348 raft_consensus.cc:504] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.160385 19348 leader_election.cc:288] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa32276c1a264f1c8b76981b873e8384 (127.1.217.65:34301), 3d6fbb68aa4b4364ac59d2aff2bec43d (127.1.217.66:46631)
I20260926 07:05:31.164080 19339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.160597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:47081 (local address 127.1.217.66:46631)
0926 07:05:31.160659 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.160661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.160671 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:31.160760 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.160780 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.160790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.160858 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.161072 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.161656 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163086 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.163676 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163756 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.163765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.163970 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.164009 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:05:31.164080 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.160637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:58287 (local address 127.1.217.65:34301)
0926 07:05:31.160728 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.160730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.160900 (+ 170us) server_negotiation.cc:407] Connection header received
0926 07:05:31.160936 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.160937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.160946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.161010 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.161207 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.161683 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163087 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.163671 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163756 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.163765 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.163970 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.164012 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:05:31.164139 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.160543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:46631 (local address 127.1.217.67:47081)
0926 07:05:31.160608 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.160622 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.160686 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.160917 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.160919 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.160921 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.161002 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.161004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.161729 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.161730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.162978 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.162981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.163785 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.163888 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.163890 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.163954 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.164008 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":10}
I20260926 07:05:31.164148 19351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.160549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34301 (local address 127.1.217.67:58287)
0926 07:05:31.160825 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.160835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.160889 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.161035 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.161035 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.161038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.161111 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.161113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.161804 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.161805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.162978 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.162981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.163785 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.163786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.163879 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.163883 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.163954 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.164008 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":63,"thread_start_us":131,"threads_started":1}
I20260926 07:05:31.164492 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ca9d825b9e4b0b8b446557b2b38109" candidate_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa32276c1a264f1c8b76981b873e8384" is_pre_election: true
I20260926 07:05:31.164671 19011 raft_consensus.cc:2454] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0755ec8a7b40e0aa19a6057f4918f2 in term 0.
I20260926 07:05:31.164638 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ca9d825b9e4b0b8b446557b2b38109" candidate_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" is_pre_election: true
I20260926 07:05:31.164788 19145 raft_consensus.cc:2454] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b0755ec8a7b40e0aa19a6057f4918f2 in term 0.
I20260926 07:05:31.164914 19214 leader_election.cc:302] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0755ec8a7b40e0aa19a6057f4918f2, aa32276c1a264f1c8b76981b873e8384; no voters:
W20260926 07:05:31.165112 19192 tablet.cc:2369] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:31.165307 19348 raft_consensus.cc:2781] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:31.165359 19348 raft_consensus.cc:482] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:31.165373 19348 raft_consensus.cc:3037] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:31.166040 19348 raft_consensus.cc:504] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.166178 19348 leader_election.cc:288] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [CANDIDATE]: Term 1 election: Requested vote from peers aa32276c1a264f1c8b76981b873e8384 (127.1.217.65:34301), 3d6fbb68aa4b4364ac59d2aff2bec43d (127.1.217.66:46631)
I20260926 07:05:31.166343 19011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ca9d825b9e4b0b8b446557b2b38109" candidate_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa32276c1a264f1c8b76981b873e8384"
I20260926 07:05:31.166352 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ca9d825b9e4b0b8b446557b2b38109" candidate_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d"
I20260926 07:05:31.166430 19145 raft_consensus.cc:3037] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:31.166431 19011 raft_consensus.cc:3037] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:31.167317 19011 raft_consensus.cc:2454] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0755ec8a7b40e0aa19a6057f4918f2 in term 1.
I20260926 07:05:31.167317 19145 raft_consensus.cc:2454] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b0755ec8a7b40e0aa19a6057f4918f2 in term 1.
I20260926 07:05:31.167511 19214 leader_election.cc:302] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b0755ec8a7b40e0aa19a6057f4918f2, aa32276c1a264f1c8b76981b873e8384; no voters:
I20260926 07:05:31.167629 19348 raft_consensus.cc:2781] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:31.167800 19348 raft_consensus.cc:686] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b0755ec8a7b40e0aa19a6057f4918f2, State: Running, Role: LEADER
I20260926 07:05:31.167943 19348 consensus_queue.cc:237] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } }
I20260926 07:05:31.168748 18865 catalog_manager.cc:5495] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b0755ec8a7b40e0aa19a6057f4918f2 (127.1.217.67). New cstate: current_term: 1 leader_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b0755ec8a7b40e0aa19a6057f4918f2" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 } health_report { overall_health: UNKNOWN } } }
I20260926 07:05:31.173941 19340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.171636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.1.217.67:46339)
0926 07:05:31.171692 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.171694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.171703 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:31.171731 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.171732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.171752 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.171823 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:31.171953 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.172406 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.173230 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.173417 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.173483 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.173647 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:31.173733 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:31.173819 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.173839 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.173857 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:05:31.175544 19145 raft_consensus.cc:1261] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 1 FOLLOWER]: Refusing update from remote peer 4b0755ec8a7b40e0aa19a6057f4918f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:31.175544 19011 raft_consensus.cc:1261] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 1 FOLLOWER]: Refusing update from remote peer 4b0755ec8a7b40e0aa19a6057f4918f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:31.175835 19352 consensus_queue.cc:1035] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa32276c1a264f1c8b76981b873e8384" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:31.175941 19348 consensus_queue.cc:1035] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d6fbb68aa4b4364ac59d2aff2bec43d" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 46631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:31.178058 19355 mvcc.cc:205] Tried to move back new op lower bound from 7333504332492496896 to 7333504332464635904. Current Snapshot: MvccSnapshot[applied={T|T < 7333504332492496896}]
I20260926 07:05:31.178797 19356 mvcc.cc:205] Tried to move back new op lower bound from 7333504332492496896 to 7333504332464635904. Current Snapshot: MvccSnapshot[applied={T|T < 7333504332492496896}]
I20260926 07:05:31.180871 18866 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:31.180967 18866 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:31.181734 18866 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9c7e337f15d475e840db22c2f14e01b: Sending DeleteTablet for 3 replicas of tablet 47ca9d825b9e4b0b8b446557b2b38109
I20260926 07:05:31.182039 19259 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:31 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:48390
I20260926 07:05:31.182057 18991 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:31 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:35834
I20260926 07:05:31.182356 19125 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ca9d825b9e4b0b8b446557b2b38109 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:31 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:58230
I20260926 07:05:31.182447 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18928
I20260926 07:05:31.182561 19366 tablet_replica.cc:330] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384: stopping tablet replica
I20260926 07:05:31.182651 19366 raft_consensus.cc:2229] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:31.182735 19366 raft_consensus.cc:2258] T 47ca9d825b9e4b0b8b446557b2b38109 P aa32276c1a264f1c8b76981b873e8384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:31.183043 19367 tablet_replica.cc:330] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: stopping tablet replica
I20260926 07:05:31.183135 19367 raft_consensus.cc:2229] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:31.183224 19367 raft_consensus.cc:2258] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:31.183609 19367 ts_tablet_manager.cc:1905] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:31.185309 19367 ts_tablet_manager.cc:1918] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:31.185405 19367 log.cc:1196] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/47ca9d825b9e4b0b8b446557b2b38109
I20260926 07:05:31.185667 19367 ts_tablet_manager.cc:1939] T 47ca9d825b9e4b0b8b446557b2b38109 P 3d6fbb68aa4b4364ac59d2aff2bec43d: Deleting consensus metadata
I20260926 07:05:31.186074 18851 catalog_manager.cc:4841] TS 3d6fbb68aa4b4364ac59d2aff2bec43d (127.1.217.66:46631): tablet 47ca9d825b9e4b0b8b446557b2b38109 (table test-table [id=406f6ca3bcc94a638ba2168374e0b3cd]) successfully deleted
I20260926 07:05:31.190609 19365 tablet_replica.cc:330] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: stopping tablet replica
W20260926 07:05:31.190634 18852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:34301 (error 108)
I20260926 07:05:31.190718 19365 raft_consensus.cc:2229] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:05:31.190743 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19062
I20260926 07:05:31.190845 19365 raft_consensus.cc:2258] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:31.191470 19365 ts_tablet_manager.cc:1905] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:31.191668 19337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.191525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:34301 (local address 127.0.0.1:35846)
0926 07:05:31.191594 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.191631 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 07:05:31.191895 18852 catalog_manager.cc:4584] TS aa32276c1a264f1c8b76981b873e8384 (127.1.217.65:34301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47ca9d825b9e4b0b8b446557b2b38109: Network error: Client connection negotiation failed: client connection to 127.1.217.65:34301: connect: Connection refused (error 111)
I20260926 07:05:31.193421 19365 ts_tablet_manager.cc:1918] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:31.193536 19365 log.cc:1196] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/47ca9d825b9e4b0b8b446557b2b38109
I20260926 07:05:31.193986 19365 ts_tablet_manager.cc:1939] T 47ca9d825b9e4b0b8b446557b2b38109 P 4b0755ec8a7b40e0aa19a6057f4918f2: Deleting consensus metadata
I20260926 07:05:31.194417 18853 catalog_manager.cc:4841] TS 4b0755ec8a7b40e0aa19a6057f4918f2 (127.1.217.67:46339): tablet 47ca9d825b9e4b0b8b446557b2b38109 (table test-table [id=406f6ca3bcc94a638ba2168374e0b3cd]) successfully deleted
I20260926 07:05:31.196740 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19196
I20260926 07:05:31.201005 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 18834
2026-09-26T07:05:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1931 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T07:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:31Z Disabled control of system clock
I20260926 07:05:31.229122 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42931
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:31.313861 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:31.314100 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:31.314123 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:31.315325 19372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:31.315362 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:31.315379 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:31.315393 19372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:31.315407 19372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:31.316877 19372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42931
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:31.317081 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:31.317348 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:31.322111 19377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.322122 19378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.322229 19380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.322376 19372 server_base.cc:1034] running on GCE node
I20260926 07:05:31.322621 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:31.322929 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:31.324116 19372 hybrid_clock.cc:648] HybridClock initialized: now 1790406331324101 us; error 28 us; skew 500 ppm
I20260926 07:05:31.325363 19372 webserver.cc:460] Webserver started at http://127.1.217.126:44471/ using document root <none> and password file <none>
I20260926 07:05:31.325618 19372 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:31.325683 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:31.325829 19372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:31.326895 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "6fdc12ff3722479ea3f567304f20e65e"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "ceded118df75ade60615ac13604c8d5f"
server_key_iv: "9132c7e3de18a4785fa413dc39ca12a4"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.327251 19372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "6fdc12ff3722479ea3f567304f20e65e"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "ceded118df75ade60615ac13604c8d5f"
server_key_iv: "9132c7e3de18a4785fa413dc39ca12a4"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.328517 19372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:31.329444 19386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.329617 19372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:31.329720 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "6fdc12ff3722479ea3f567304f20e65e"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "ceded118df75ade60615ac13604c8d5f"
server_key_iv: "9132c7e3de18a4785fa413dc39ca12a4"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.329807 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:31.339143 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:31.339407 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:31.339546 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:31.343350 19372 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42931
I20260926 07:05:31.343379 19438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42931 every 8 connection(s)
I20260926 07:05:31.343688 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:31.344635 19439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.346844 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: Bootstrap starting.
I20260926 07:05:31.347393 19439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.347631 19439 log.cc:824] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.348294 19439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: No bootstrap required, opened a new log
I20260926 07:05:31.349849 19439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } }
I20260926 07:05:31.349992 19439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.350013 19439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdc12ff3722479ea3f567304f20e65e, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.350142 19439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } }
I20260926 07:05:31.350211 19439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:31.350234 19439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:31.350260 19439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:31.350713 19439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } }
I20260926 07:05:31.350790 19439 leader_election.cc:302] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fdc12ff3722479ea3f567304f20e65e; no voters:
I20260926 07:05:31.350939 19439 leader_election.cc:288] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:31.350970 19442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:31.351152 19442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [term 1 LEADER]: Becoming Leader. State: Replica: 6fdc12ff3722479ea3f567304f20e65e, State: Running, Role: LEADER
I20260926 07:05:31.351239 19439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:31.351307 19442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } }
I20260926 07:05:31.351825 19443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fdc12ff3722479ea3f567304f20e65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } } }
I20260926 07:05:31.351926 19443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:31.352448 19451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:31.353044 19451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:31.353092 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19372
I20260926 07:05:31.353209 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:31.353415 1893 external_mini_cluster.cc:1428] Setting key e4f4fb32f55f87cc2c3f86394a66a775
I20260926 07:05:31.353714 19444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fdc12ff3722479ea3f567304f20e65e. Latest consensus state: current_term: 1 leader_uuid: "6fdc12ff3722479ea3f567304f20e65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fdc12ff3722479ea3f567304f20e65e" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42931 } } }
I20260926 07:05:31.354624 19444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:31.354817 19451 catalog_manager.cc:1347] Generated new cluster ID: 01bad4e006114126aea60b9a61fe940d
I20260926 07:05:31.354872 19451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:31.360148 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.353909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.1.217.126:42931)
0926 07:05:31.354096 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.354106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.354189 (+ 83us) server_negotiation.cc:407] Connection header received
0926 07:05:31.354368 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.354377 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.354477 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.355065 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.355163 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.356625 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.357797 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.358469 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.358587 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.359332 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.359337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.359338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:31.359558 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.359600 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:31.359615 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.359680 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.359759 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.359933 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260926 07:05:31.369683 19451 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:31.370232 19451 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:31.374778 19451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: Generated new TSK 0
I20260926 07:05:31.374984 19451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:31.377879 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--builtin_ntp_servers=127.1.217.84:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:31.477538 19463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:31.477727 19463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:31.477747 19463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:31.478988 19463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:31.479029 19463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:31.479054 19463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:31.480361 19463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:31.480576 19463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:31.480906 19463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:31.486608 19469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.486657 19471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.486616 19468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.486786 19463 server_base.cc:1034] running on GCE node
I20260926 07:05:31.487010 19463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:31.487248 19463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:31.488408 19463 hybrid_clock.cc:648] HybridClock initialized: now 1790406331488391 us; error 34 us; skew 500 ppm
I20260926 07:05:31.489674 19463 webserver.cc:460] Webserver started at http://127.1.217.65:36361/ using document root <none> and password file <none>
I20260926 07:05:31.489907 19463 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:31.489974 19463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:31.490113 19463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:31.491192 19463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "75cf614e28db42c6827a0708c00979f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "6aca713911bb910164ba3a0f8ac64b29"
server_key_iv: "21f37210100369587421ede09ac3a596"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.491544 19463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "75cf614e28db42c6827a0708c00979f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "6aca713911bb910164ba3a0f8ac64b29"
server_key_iv: "21f37210100369587421ede09ac3a596"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.492895 19463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:31.493712 19477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.493880 19463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:31.493978 19463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "75cf614e28db42c6827a0708c00979f2"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "6aca713911bb910164ba3a0f8ac64b29"
server_key_iv: "21f37210100369587421ede09ac3a596"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.494050 19463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:31.509019 19463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:31.509336 19463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:31.509487 19463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:31.509719 19463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:31.510041 19463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:31.510074 19463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.510097 19463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:31.510139 19463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.515815 19463 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:43537
I20260926 07:05:31.515938 19590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:43537 every 8 connection(s)
I20260926 07:05:31.516237 19463 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:05:31.521595 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.517056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:54849 (local address 127.1.217.126:42931)
0926 07:05:31.517120 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.517122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.517378 (+ 256us) server_negotiation.cc:407] Connection header received
0926 07:05:31.517906 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.517907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.517921 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.517978 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.518734 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.519238 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.520223 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.520426 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.520485 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.521374 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.521375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.521375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:31.521380 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.521393 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:31.521398 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.521443 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.521510 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.521534 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:05:31.521808 19593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.517064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42931 (local address 127.1.217.65:54849)
0926 07:05:31.517256 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.517291 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.517799 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.518055 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.518059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.518067 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:31.518605 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.518612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.519325 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.519326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.520135 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.520137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.520506 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.520507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.520620 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.521233 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:31.521233 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:31.521263 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:31.521286 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:31.521445 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:31.521447 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:31.521450 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.521500 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.521575 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":85,"threads_started":1}
I20260926 07:05:31.522306 19591 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42931
I20260926 07:05:31.522405 19591 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:31.522590 19591 heartbeater.cc:511] Master 127.1.217.126:42931 requested a full tablet report, sending...
I20260926 07:05:31.523102 19403 ts_manager.cc:194] Registered new tserver with Master: 75cf614e28db42c6827a0708c00979f2 (127.1.217.65:43537)
I20260926 07:05:31.523245 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19463
I20260926 07:05:31.523366 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:31.523568 1893 external_mini_cluster.cc:1428] Setting key 40e05b133b91bb2b4e901025a0ec6103
I20260926 07:05:31.523948 19403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.65:54849
I20260926 07:05:31.524914 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--builtin_ntp_servers=127.1.217.84:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:31.606088 19594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:31.606292 19594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:31.606314 19594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:31.607448 19594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:31.607487 19594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:31.607512 19594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:31.608889 19594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:31.609118 19594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:31.609407 19594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:31.614463 19602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.614466 19599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.614439 19600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.614674 19594 server_base.cc:1034] running on GCE node
I20260926 07:05:31.614913 19594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:31.615164 19594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:31.616325 19594 hybrid_clock.cc:648] HybridClock initialized: now 1790406331616320 us; error 35 us; skew 500 ppm
I20260926 07:05:31.617656 19594 webserver.cc:460] Webserver started at http://127.1.217.66:41653/ using document root <none> and password file <none>
I20260926 07:05:31.617918 19594 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:31.617986 19594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:31.618124 19594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:31.619192 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "fd38f322ba034d2d8a04f4071284e84b"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "a4ef47159b6db232723575b5ddd3edc5"
server_key_iv: "deff7b4bcf825d6c1cad5157b6a9e84e"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.619550 19594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "fd38f322ba034d2d8a04f4071284e84b"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "a4ef47159b6db232723575b5ddd3edc5"
server_key_iv: "deff7b4bcf825d6c1cad5157b6a9e84e"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.620862 19594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:31.621605 19608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.621795 19594 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:31.621907 19594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "fd38f322ba034d2d8a04f4071284e84b"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "a4ef47159b6db232723575b5ddd3edc5"
server_key_iv: "deff7b4bcf825d6c1cad5157b6a9e84e"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.621989 19594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:31.650861 19594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:31.651228 19594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:31.651423 19594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:31.651691 19594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:31.652073 19594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:31.652123 19594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.652166 19594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:31.652194 19594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.657923 19594 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:36831
I20260926 07:05:31.657971 19721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:36831 every 8 connection(s)
I20260926 07:05:31.658298 19594 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:05:31.659821 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19594
I20260926 07:05:31.659976 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:31.660199 1893 external_mini_cluster.cc:1428] Setting key 8ec56d3fb1479818581f5f9ff7f9c7ef
I20260926 07:05:31.661286 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--builtin_ntp_servers=127.1.217.84:33179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:05:31.664175 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.659065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:52275 (local address 127.1.217.126:42931)
0926 07:05:31.659116 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.659118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.659349 (+ 231us) server_negotiation.cc:407] Connection header received
0926 07:05:31.659911 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.659913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.659926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.659988 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.660849 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.661718 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.662788 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.662988 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.663066 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.663888 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.663890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.663890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:31.663895 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.663908 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:31.663913 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.663978 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.664067 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.664097 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:05:31.664404 19724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.659046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42931 (local address 127.1.217.66:52275)
0926 07:05:31.659228 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.659262 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.659795 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.660064 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.660068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.660078 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:31.660737 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.660743 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.661836 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.661837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.662662 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.662664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.663073 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.663075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.663210 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.663737 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:31.663737 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:31.663766 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:31.663790 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:31.663979 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:31.663985 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:31.663988 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.664071 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.664171 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260926 07:05:31.664933 19722 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42931
I20260926 07:05:31.665036 19722 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:31.665220 19722 heartbeater.cc:511] Master 127.1.217.126:42931 requested a full tablet report, sending...
I20260926 07:05:31.665634 19402 ts_manager.cc:194] Registered new tserver with Master: fd38f322ba034d2d8a04f4071284e84b (127.1.217.66:36831)
I20260926 07:05:31.666015 19402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.66:52275
W20260926 07:05:31.745895 19725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:31.746098 19725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:31.746119 19725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:31.747325 19725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:31.747370 19725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:31.747395 19725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:31.748730 19725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:31.748981 19725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:31.749289 19725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:31.754267 19733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:31.754274 19730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.754410 19725 server_base.cc:1034] running on GCE node
W20260926 07:05:31.754272 19731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:31.754673 19725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:31.754889 19725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:31.756023 19725 hybrid_clock.cc:648] HybridClock initialized: now 1790406331756012 us; error 27 us; skew 500 ppm
I20260926 07:05:31.757347 19725 webserver.cc:460] Webserver started at http://127.1.217.67:41891/ using document root <none> and password file <none>
I20260926 07:05:31.757571 19725 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:31.757617 19725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:31.757771 19725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:31.758793 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "a4358219f89b46c78127daf8de525391"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "d47586e2509d369837d25f0d5d8ab662"
server_key_iv: "cb4be8703429ca29b71530526c270883"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.759159 19725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "a4358219f89b46c78127daf8de525391"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "d47586e2509d369837d25f0d5d8ab662"
server_key_iv: "cb4be8703429ca29b71530526c270883"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.760368 19725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:05:31.761099 19739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.761260 19725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.761325 19725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "a4358219f89b46c78127daf8de525391"
format_stamp: "Formatted at 2026-09-26 07:05:31 on dist-test-slave-t1mc"
server_key: "d47586e2509d369837d25f0d5d8ab662"
server_key_iv: "cb4be8703429ca29b71530526c270883"
server_key_version: "encryptionkey@0"
I20260926 07:05:31.761406 19725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:31.790437 19725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:31.790767 19725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:31.790921 19725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:31.791179 19725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:31.791515 19725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:31.791550 19725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.791590 19725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:31.791617 19725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:31.797399 19725 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:46253
I20260926 07:05:31.797473 19852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:46253 every 8 connection(s)
I20260926 07:05:31.797797 19725 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:31.802896 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.798507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:52161 (local address 127.1.217.126:42931)
0926 07:05:31.798554 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.798556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.798792 (+ 236us) server_negotiation.cc:407] Connection header received
0926 07:05:31.799266 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.799268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.799278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.799342 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.799990 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.800487 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.801528 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.801721 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.801777 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.802557 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.802559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.802559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:31.802564 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.802578 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:31.802582 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.802651 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.802766 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.802800 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:05:31.803092 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.798483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42931 (local address 127.1.217.67:52161)
0926 07:05:31.798665 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.798705 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.799178 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.799411 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.799415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.799423 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:31.799918 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.799922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.800568 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.800569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.801437 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.801440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.801795 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.801796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.801906 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.802421 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:31.802421 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:31.802450 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:31.802474 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:31.802662 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:31.802666 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:31.802669 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.802762 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.802879 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 07:05:31.803599 19853 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42931
I20260926 07:05:31.803695 19853 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:31.803901 19853 heartbeater.cc:511] Master 127.1.217.126:42931 requested a full tablet report, sending...
I20260926 07:05:31.804335 19402 ts_manager.cc:194] Registered new tserver with Master: a4358219f89b46c78127daf8de525391 (127.1.217.67:46253)
I20260926 07:05:31.804704 19402 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.67:52161
I20260926 07:05:31.807325 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19725
I20260926 07:05:31.807418 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:31.807574 1893 external_mini_cluster.cc:1428] Setting key fe5facc87ab71cb21df8752777a09c48
I20260926 07:05:31.808470 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:31.812600 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.809841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.1.217.126:42931)
0926 07:05:31.809896 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.809897 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.810041 (+ 144us) server_negotiation.cc:407] Connection header received
0926 07:05:31.810086 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.810087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.810098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.810155 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:31.810283 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.810818 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.811548 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.811728 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.811780 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.812400 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:31.812401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:31.812401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:31.812411 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:31.812424 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:31.812427 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:31.812470 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.812519 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.812541 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:05:31.814174 19402 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:31.815723 19402 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:31.822158 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.818260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.1.217.65:43537)
0926 07:05:31.818462 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.818479 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.818489 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:31.818563 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.818564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.818576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.818677 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.818913 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.820067 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821252 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.821739 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821835 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.821933 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.822022 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.822061 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":86,"threads_started":1}
I20260926 07:05:31.822146 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.818010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43537 (local address 127.0.0.1:42466)
0926 07:05:31.818423 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.818431 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.818491 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.818748 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.818749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.818752 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.818829 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.818832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.820161 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.820162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.821175 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.821177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.821838 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821838 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.821910 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.821923 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.822007 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.822062 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":11,"thread_start_us":87,"threads_started":1}
I20260926 07:05:31.822351 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.818080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.1.217.67:46253)
0926 07:05:31.818470 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.818475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.818485 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:31.818528 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.818530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.818547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.818625 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.818933 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.819666 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821046 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.821790 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821860 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.821945 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.822197 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.822240 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":77,"threads_started":1}
I20260926 07:05:31.822400 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.817999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:46253 (local address 127.0.0.1:53654)
0926 07:05:31.818196 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.818239 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.818305 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.818745 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.818748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.818750 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.818826 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.818828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.819760 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.819761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.820961 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.820964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.821983 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.822045 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.822050 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.822098 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.822140 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260926 07:05:31.823012 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.818027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36831 (local address 127.0.0.1:34890)
0926 07:05:31.818606 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.818614 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.818657 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.818920 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.818921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.818923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.818957 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.818958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.820200 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.820201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.821172 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.821174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.822207 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.822208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.822800 (+ 592us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.822804 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.822881 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.822923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":102,"threads_started":1}
I20260926 07:05:31.822928 19525 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 (DEFAULT_TABLE table=test-table [id=8901fa5de64243ec8be7634ae7d665be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.823067 19787 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 (DEFAULT_TABLE table=test-table [id=8901fa5de64243ec8be7634ae7d665be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.823319 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.818253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.1.217.66:36831)
0926 07:05:31.818415 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.818421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.818720 (+ 299us) server_negotiation.cc:407] Connection header received
0926 07:05:31.818767 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.818769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.818784 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.818853 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.819039 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.820113 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.821263 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.822020 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.822086 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.822164 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.823185 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.823227 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260926 07:05:31.823323 19525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab89923f7fc4f308e9de04a9cacbf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.823338 19787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab89923f7fc4f308e9de04a9cacbf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.823901 19656 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 (DEFAULT_TABLE table=test-table [id=8901fa5de64243ec8be7634ae7d665be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:31.824271 19656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab89923f7fc4f308e9de04a9cacbf83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:31.826934 19868 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Bootstrap starting.
I20260926 07:05:31.827169 19867 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Bootstrap starting.
I20260926 07:05:31.827363 19869 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Bootstrap starting.
I20260926 07:05:31.827896 19867 tablet_bootstrap.cc:654] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.827950 19869 tablet_bootstrap.cc:654] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.828262 19869 log.cc:824] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.828262 19867 log.cc:824] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.829041 19867 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: No bootstrap required, opened a new log
I20260926 07:05:31.829083 19868 tablet_bootstrap.cc:654] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:31.829123 19867 ts_tablet_manager.cc:1397] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:31.829265 19869 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: No bootstrap required, opened a new log
I20260926 07:05:31.829339 19869 ts_tablet_manager.cc:1397] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:31.829519 19868 log.cc:824] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:31.830309 19868 tablet_bootstrap.cc:492] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: No bootstrap required, opened a new log
I20260926 07:05:31.830381 19868 ts_tablet_manager.cc:1397] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 07:05:31.831458 19867 raft_consensus.cc:348] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.831620 19867 raft_consensus.cc:374] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.831664 19867 raft_consensus.cc:729] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75cf614e28db42c6827a0708c00979f2, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.831782 19867 consensus_queue.cc:260] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.831934 19869 raft_consensus.cc:348] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.832082 19591 heartbeater.cc:503] Master 127.1.217.126:42931 was elected leader, sending a full tablet report...
I20260926 07:05:31.832114 19869 raft_consensus.cc:374] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.832157 19869 raft_consensus.cc:729] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd38f322ba034d2d8a04f4071284e84b, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.832271 19869 consensus_queue.cc:260] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.832391 19867 ts_tablet_manager.cc:1428] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:05:31.832552 19869 ts_tablet_manager.cc:1428] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:05:31.832690 19722 heartbeater.cc:503] Master 127.1.217.126:42931 was elected leader, sending a full tablet report...
I20260926 07:05:31.833659 19868 raft_consensus.cc:348] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.833813 19868 raft_consensus.cc:374] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:31.833837 19868 raft_consensus.cc:729] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4358219f89b46c78127daf8de525391, State: Initialized, Role: FOLLOWER
I20260926 07:05:31.833966 19868 consensus_queue.cc:260] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.834214 19868 ts_tablet_manager.cc:1428] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 07:05:31.834239 19853 heartbeater.cc:503] Master 127.1.217.126:42931 was elected leader, sending a full tablet report...
W20260926 07:05:31.908902 19723 tablet.cc:2369] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:31.995877 19875 raft_consensus.cc:482] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:31.996016 19875 raft_consensus.cc:504] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:31.996399 19875 leader_election.cc:288] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75cf614e28db42c6827a0708c00979f2 (127.1.217.65:43537), fd38f322ba034d2d8a04f4071284e84b (127.1.217.66:36831)
I20260926 07:05:31.999353 19866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.996719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:49895 (local address 127.1.217.65:43537)
0926 07:05:31.996806 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.996808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.996954 (+ 146us) server_negotiation.cc:407] Connection header received
0926 07:05:31.997018 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.997019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.997033 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.997096 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.997238 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.997730 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.998658 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.998998 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.999077 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.999085 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.999237 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.999272 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 07:05:31.999387 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.996716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:58307 (local address 127.1.217.66:36831)
0926 07:05:31.996787 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:31.996789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:31.996799 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:31.996839 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:31.996840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:31.996851 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:31.996896 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:31.997087 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.997627 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.998655 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.999030 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.999119 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.999126 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:31.999268 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 07:05:31.999307 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:05:31.999395 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.996616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:36831 (local address 127.1.217.67:58307)
0926 07:05:31.996684 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.996698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.996750 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.996940 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.996941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.996942 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.997010 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.997012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.997713 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.997714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.998539 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.998542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.999119 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.999120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.999195 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.999197 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.999262 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.999304 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 07:05:31.999540 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:31.996618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43537 (local address 127.1.217.67:49895)
0926 07:05:31.996868 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:31.996876 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:31.996954 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:31.997097 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:31.997098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:31.997100 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:31.997173 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.997175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.997806 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.997807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.998572 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:31.998574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:31.999076 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:31.999077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:31.999156 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:31.999160 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:31.999229 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 07:05:31.999271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":76,"thread_start_us":114,"threads_started":1}
I20260926 07:05:31.999859 19676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab89923f7fc4f308e9de04a9cacbf83" candidate_uuid: "a4358219f89b46c78127daf8de525391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd38f322ba034d2d8a04f4071284e84b" is_pre_election: true
I20260926 07:05:31.999859 19545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab89923f7fc4f308e9de04a9cacbf83" candidate_uuid: "a4358219f89b46c78127daf8de525391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf614e28db42c6827a0708c00979f2" is_pre_election: true
I20260926 07:05:32.000051 19545 raft_consensus.cc:2454] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4358219f89b46c78127daf8de525391 in term 0.
I20260926 07:05:32.000047 19676 raft_consensus.cc:2454] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4358219f89b46c78127daf8de525391 in term 0.
I20260926 07:05:32.000289 19743 leader_election.cc:302] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75cf614e28db42c6827a0708c00979f2, a4358219f89b46c78127daf8de525391; no voters:
I20260926 07:05:32.000440 19875 raft_consensus.cc:2781] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:32.000496 19875 raft_consensus.cc:482] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:32.000520 19875 raft_consensus.cc:3037] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:32.001224 19875 raft_consensus.cc:504] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:32.001370 19875 leader_election.cc:288] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [CANDIDATE]: Term 1 election: Requested vote from peers 75cf614e28db42c6827a0708c00979f2 (127.1.217.65:43537), fd38f322ba034d2d8a04f4071284e84b (127.1.217.66:36831)
I20260926 07:05:32.001530 19545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab89923f7fc4f308e9de04a9cacbf83" candidate_uuid: "a4358219f89b46c78127daf8de525391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75cf614e28db42c6827a0708c00979f2"
I20260926 07:05:32.001549 19676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab89923f7fc4f308e9de04a9cacbf83" candidate_uuid: "a4358219f89b46c78127daf8de525391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd38f322ba034d2d8a04f4071284e84b"
I20260926 07:05:32.001633 19676 raft_consensus.cc:3037] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:32.001624 19545 raft_consensus.cc:3037] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:32.002589 19545 raft_consensus.cc:2454] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4358219f89b46c78127daf8de525391 in term 1.
I20260926 07:05:32.002589 19676 raft_consensus.cc:2454] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4358219f89b46c78127daf8de525391 in term 1.
I20260926 07:05:32.002792 19740 leader_election.cc:302] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4358219f89b46c78127daf8de525391, fd38f322ba034d2d8a04f4071284e84b; no voters:
I20260926 07:05:32.002956 19875 raft_consensus.cc:2781] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:32.003167 19875 raft_consensus.cc:686] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 1 LEADER]: Becoming Leader. State: Replica: a4358219f89b46c78127daf8de525391, State: Running, Role: LEADER
I20260926 07:05:32.003389 19875 consensus_queue.cc:237] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } }
I20260926 07:05:32.004267 19402 catalog_manager.cc:5495] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4358219f89b46c78127daf8de525391 (127.1.217.67). New cstate: current_term: 1 leader_uuid: "a4358219f89b46c78127daf8de525391" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4358219f89b46c78127daf8de525391" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 } health_report { overall_health: UNKNOWN } } }
W20260926 07:05:32.016993 19592 tablet.cc:2369] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:05:32.048336 19854 tablet.cc:2369] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:32.049835 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:32.047321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.1.217.67:46253)
0926 07:05:32.047387 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:32.047389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:32.047410 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:05:32.047471 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:32.047472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:32.047492 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:32.047569 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:32.047703 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:32.048204 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:32.049093 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:32.049288 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:32.049361 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:32.049500 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:32.049598 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:32.049688 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:32.049713 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 07:05:32.049731 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:05:32.051646 19545 raft_consensus.cc:1261] T 1ab89923f7fc4f308e9de04a9cacbf83 P 75cf614e28db42c6827a0708c00979f2 [term 1 FOLLOWER]: Refusing update from remote peer a4358219f89b46c78127daf8de525391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:32.051646 19676 raft_consensus.cc:1261] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 1 FOLLOWER]: Refusing update from remote peer a4358219f89b46c78127daf8de525391: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:32.052021 19877 consensus_queue.cc:1035] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd38f322ba034d2d8a04f4071284e84b" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 36831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:32.052175 19875 consensus_queue.cc:1035] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75cf614e28db42c6827a0708c00979f2" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 43537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:32.055184 19881 mvcc.cc:205] Tried to move back new op lower bound from 7333504336080441344 to 7333504335886630912. Current Snapshot: MvccSnapshot[applied={T|T < 7333504336080441344}]
I20260926 07:05:32.056030 19882 mvcc.cc:205] Tried to move back new op lower bound from 7333504336080441344 to 7333504335886630912. Current Snapshot: MvccSnapshot[applied={T|T < 7333504336080441344 or (T in {7333504336080441344})}]
I20260926 07:05:32.056931 19403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:32.057047 19403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:32.057816 19403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6fdc12ff3722479ea3f567304f20e65e: Sending DeleteTablet for 3 replicas of tablet 1ab89923f7fc4f308e9de04a9cacbf83
I20260926 07:05:32.058162 19787 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:32 UTC) from {username='slave'} at 127.0.0.1:53654
I20260926 07:05:32.058207 19656 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:32 UTC) from {username='slave'} at 127.0.0.1:34890
I20260926 07:05:32.058205 19525 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab89923f7fc4f308e9de04a9cacbf83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:32 UTC) from {username='slave'} at 127.0.0.1:42466
I20260926 07:05:32.058385 19890 tablet_replica.cc:330] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: stopping tablet replica
I20260926 07:05:32.058501 19890 raft_consensus.cc:2229] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:05:32.058542 19891 tablet_replica.cc:330] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: stopping tablet replica
I20260926 07:05:32.058601 19890 raft_consensus.cc:2258] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:32.058621 19891 raft_consensus.cc:2229] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:32.058717 19891 raft_consensus.cc:2258] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:32.058724 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19463
I20260926 07:05:32.059098 19890 ts_tablet_manager.cc:1905] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:32.059801 19891 ts_tablet_manager.cc:1905] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:32.061633 19891 ts_tablet_manager.cc:1918] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:32.061717 19891 log.cc:1196] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/1ab89923f7fc4f308e9de04a9cacbf83
I20260926 07:05:32.061972 19891 ts_tablet_manager.cc:1939] T 1ab89923f7fc4f308e9de04a9cacbf83 P fd38f322ba034d2d8a04f4071284e84b: Deleting consensus metadata
I20260926 07:05:32.062395 19387 catalog_manager.cc:4841] TS fd38f322ba034d2d8a04f4071284e84b (127.1.217.66:36831): tablet 1ab89923f7fc4f308e9de04a9cacbf83 (table test-table [id=8901fa5de64243ec8be7634ae7d665be]) successfully deleted
W20260926 07:05:32.065522 19390 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:43537 (error 108)
I20260926 07:05:32.065750 19890 ts_tablet_manager.cc:1918] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:32.065837 19890 log.cc:1196] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/1ab89923f7fc4f308e9de04a9cacbf83
I20260926 07:05:32.066023 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:32.065882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:43537 (local address 127.0.0.1:42470)
0926 07:05:32.065942 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:32.065983 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:43537: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:05:32.066138 19890 ts_tablet_manager.cc:1939] T 1ab89923f7fc4f308e9de04a9cacbf83 P a4358219f89b46c78127daf8de525391: Deleting consensus metadata
W20260926 07:05:32.066148 19390 catalog_manager.cc:4584] TS 75cf614e28db42c6827a0708c00979f2 (127.1.217.65:43537): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ab89923f7fc4f308e9de04a9cacbf83: Network error: Client connection negotiation failed: client connection to 127.1.217.65:43537: connect: Connection refused (error 111)
I20260926 07:05:32.066593 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19594
I20260926 07:05:32.066625 19389 catalog_manager.cc:4841] TS a4358219f89b46c78127daf8de525391 (127.1.217.67:46253): tablet 1ab89923f7fc4f308e9de04a9cacbf83 (table test-table [id=8901fa5de64243ec8be7634ae7d665be]) successfully deleted
I20260926 07:05:32.072530 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19725
I20260926 07:05:32.077685 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19372
2026-09-26T07:05:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (875 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:32 dist-test-slave-t1mc krb5kdc[19896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:32 dist-test-slave-t1mc krb5kdc[19896](info): set up 2 sockets
Sep 26 07:05:32 dist-test-slave-t1mc krb5kdc[19896](info): commencing operation
krb5kdc: starting...
W20260926 07:05:33.128562 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:33.248525 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34985
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:33.328680 19909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:33.328948 19909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:33.328989 19909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:33.330163 19909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:33.330230 19909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:33.330255 19909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:33.330276 19909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:33.330303 19909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:33.331759 19909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:33.332041 19909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:33.332262 19909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:33.337172 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.337227 19915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.337186 19914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:33.337337 19909 server_base.cc:1034] running on GCE node
I20260926 07:05:33.337632 19909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:33.337939 19909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:33.339097 19909 hybrid_clock.cc:648] HybridClock initialized: now 1790406333339084 us; error 29 us; skew 500 ppm
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:33.341528 19909 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:33.342092 19909 webserver.cc:460] Webserver started at http://127.1.217.126:36023/ using document root <none> and password file <none>
I20260926 07:05:33.342356 19909 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:33.342417 19909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:33.342558 19909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:33.343676 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "02fac1a0594d43dba2b0fb16a279c65b"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "0247504d3b304f5e38775e0eab2ca81a"
server_key_iv: "2df704186cdb9ea082b97c6cffb02914"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.344079 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "02fac1a0594d43dba2b0fb16a279c65b"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "0247504d3b304f5e38775e0eab2ca81a"
server_key_iv: "2df704186cdb9ea082b97c6cffb02914"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.345508 19909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:33.346274 19924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.346449 19909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:33.346551 19909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "02fac1a0594d43dba2b0fb16a279c65b"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "0247504d3b304f5e38775e0eab2ca81a"
server_key_iv: "2df704186cdb9ea082b97c6cffb02914"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.346622 19909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:33.357913 19909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:33.359746 19909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:33.359896 19909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:33.363997 19909 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34985
I20260926 07:05:33.364063 19976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34985 every 8 connection(s)
I20260926 07:05:33.364395 19909 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:33.365283 19977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:33.367447 19977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: Bootstrap starting.
I20260926 07:05:33.367995 19977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:33.368222 19977 log.cc:824] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:33.368937 19977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: No bootstrap required, opened a new log
I20260926 07:05:33.370479 19977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } }
I20260926 07:05:33.370610 19977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:33.370632 19977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fac1a0594d43dba2b0fb16a279c65b, State: Initialized, Role: FOLLOWER
I20260926 07:05:33.370728 19977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } }
I20260926 07:05:33.370777 19977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:33.370797 19977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:33.370827 19977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:33.371287 19977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } }
I20260926 07:05:33.371363 19977 leader_election.cc:302] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02fac1a0594d43dba2b0fb16a279c65b; no voters:
I20260926 07:05:33.371481 19977 leader_election.cc:288] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:33.371562 19980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:33.371735 19980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [term 1 LEADER]: Becoming Leader. State: Replica: 02fac1a0594d43dba2b0fb16a279c65b, State: Running, Role: LEADER
I20260926 07:05:33.371754 19977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:33.371824 19980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } }
I20260926 07:05:33.372212 19982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02fac1a0594d43dba2b0fb16a279c65b. Latest consensus state: current_term: 1 leader_uuid: "02fac1a0594d43dba2b0fb16a279c65b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } } }
I20260926 07:05:33.372298 19982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:33.372438 19981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02fac1a0594d43dba2b0fb16a279c65b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02fac1a0594d43dba2b0fb16a279c65b" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34985 } } }
I20260926 07:05:33.372555 19981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:33.373239 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 19909
I20260926 07:05:33.373364 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:33.373575 1893 external_mini_cluster.cc:1428] Setting key 286d7a67111a6574125d742481068230
W20260926 07:05:33.373734 19996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:33.373788 19996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:33.373845 19989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:33.374006 19989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:33.375495 19989 catalog_manager.cc:1347] Generated new cluster ID: 9e8fe1d4bcba4ad48bd39e4539fe896b
I20260926 07:05:33.375542 19989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:33.383363 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.374042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.1.217.126:34985)
0926 07:05:33.374189 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.374198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.374251 (+ 53us) server_negotiation.cc:407] Connection header received
0926 07:05:33.374502 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.374508 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.374554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.374688 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:33.375503 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.376419 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.377264 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.377673 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.377807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.380028 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:33.380031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:33.380032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:33.380043 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:33.381792 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.382214 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.382217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.382218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.382321 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.382523 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.382524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.382524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.382767 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:33.382938 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.383032 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.383171 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260926 07:05:33.398530 19989 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:33.399334 19989 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:33.409387 19989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: Generated new TSK 0
I20260926 07:05:33.409627 19989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:05:33.437675 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:33.537834 20003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:33.538036 20003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:33.538069 20003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:33.539232 20003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:33.539273 20003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:33.539327 20003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:33.540756 20003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:33.541035 20003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:33.541275 20003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:33.546231 20011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.546252 20009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.546249 20008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:33.546546 20003 server_base.cc:1034] running on GCE node
I20260926 07:05:33.546761 20003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:33.546995 20003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:33.548133 20003 hybrid_clock.cc:648] HybridClock initialized: now 1790406333548112 us; error 36 us; skew 500 ppm
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:33.550745 20003 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:05:33.551123 20003 webserver.cc:460] Webserver started at http://127.1.217.65:33931/ using document root <none> and password file <none>
I20260926 07:05:33.551303 20003 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:33.551342 20003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:33.551429 20003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:33.552470 20003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "56cdd6aa00eb479db5caca843ea31cca"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "e8ea72e44f32b5c6a34e9419a3f6d0e2"
server_key_iv: "a0a009921b32c29d5d8beea81f26670e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.552850 20003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "56cdd6aa00eb479db5caca843ea31cca"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "e8ea72e44f32b5c6a34e9419a3f6d0e2"
server_key_iv: "a0a009921b32c29d5d8beea81f26670e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.554077 20003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:33.554749 20018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.554893 20003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:33.554971 20003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "56cdd6aa00eb479db5caca843ea31cca"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "e8ea72e44f32b5c6a34e9419a3f6d0e2"
server_key_iv: "a0a009921b32c29d5d8beea81f26670e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.555025 20003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:33.566892 20003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:33.569160 20003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:33.569269 20003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:33.569446 20003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:33.569684 20003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:33.569713 20003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.569733 20003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:33.569746 20003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.575704 20003 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:37047
I20260926 07:05:33.575779 20131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:37047 every 8 connection(s)
I20260926 07:05:33.576090 20003 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:33.582944 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20003
I20260926 07:05:33.583058 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:33.583280 1893 external_mini_cluster.cc:1428] Setting key c2c058ce65189fec8964be3389dcfac8
I20260926 07:05:33.585701 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.576822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:39463 (local address 127.1.217.126:34985)
0926 07:05:33.576875 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.576877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.577085 (+ 208us) server_negotiation.cc:407] Connection header received
0926 07:05:33.577556 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.577557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.577571 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.577620 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:33.578515 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.579038 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.580056 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.580269 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.580333 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.582462 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:33.582464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:33.582464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:33.582470 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:33.584160 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.584622 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.584623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.584624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.584672 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.584961 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.584962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.584962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.585086 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:33.585136 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.585481 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.585576 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:05:33.585732 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.576770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34985 (local address 127.1.217.65:39463)
0926 07:05:33.576975 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.577012 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.577470 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.577680 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.577684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.578240 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:33.578432 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.578437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.579130 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.579131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.579903 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.579905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.580347 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.580348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.580444 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.580982 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:33.580990 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:33.582379 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:33.584229 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.584235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.584237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.584552 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.584731 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.584732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.584732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.584894 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.585145 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:33.585151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:33.585318 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.585432 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.585579 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260926 07:05:33.586220 20132 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34985
I20260926 07:05:33.586339 20132 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:33.586524 20132 heartbeater.cc:511] Master 127.1.217.126:34985 requested a full tablet report, sending...
I20260926 07:05:33.587045 19941 ts_manager.cc:194] Registered new tserver with Master: 56cdd6aa00eb479db5caca843ea31cca (127.1.217.65:37047)
I20260926 07:05:33.588021 19941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:39463
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:05:33.607686 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:33.688699 20137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:33.688922 20137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:33.688946 20137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:33.690026 20137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:33.690063 20137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:33.690089 20137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:33.691656 20137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:33.691883 20137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:33.692112 20137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:33.697019 20145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.697029 20143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.697019 20142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:33.697217 20137 server_base.cc:1034] running on GCE node
I20260926 07:05:33.697435 20137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:33.697675 20137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:33.698835 20137 hybrid_clock.cc:648] HybridClock initialized: now 1790406333698830 us; error 41 us; skew 500 ppm
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:33.701303 20137 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:05:33.701768 20137 webserver.cc:460] Webserver started at http://127.1.217.66:44039/ using document root <none> and password file <none>
I20260926 07:05:33.702011 20137 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:33.702075 20137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:33.702215 20137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:33.703251 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "1d1f5979ba7c4a479ef184e59b8221f0"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "947d3551667bda966e60a0375ef6b7e4"
server_key_iv: "c4cc263fda9d33ed3359c354cedf772e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.703613 20137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "1d1f5979ba7c4a479ef184e59b8221f0"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "947d3551667bda966e60a0375ef6b7e4"
server_key_iv: "c4cc263fda9d33ed3359c354cedf772e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.705006 20137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:05:33.705715 20152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.705879 20137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:33.705974 20137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "1d1f5979ba7c4a479ef184e59b8221f0"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "947d3551667bda966e60a0375ef6b7e4"
server_key_iv: "c4cc263fda9d33ed3359c354cedf772e"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.706055 20137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:33.715894 20137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:33.717861 20137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:33.718012 20137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:33.718240 20137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:33.718531 20137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:33.718581 20137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.718622 20137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:33.718649 20137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.724515 20137 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:39415
I20260926 07:05:33.724560 20265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:39415 every 8 connection(s)
I20260926 07:05:33.724939 20137 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:33.732496 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20137
I20260926 07:05:33.732611 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:33.732849 1893 external_mini_cluster.cc:1428] Setting key be571f7b4c51f0bc444a8a1d74dc9dce
I20260926 07:05:33.734455 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.725637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:50307 (local address 127.1.217.126:34985)
0926 07:05:33.725700 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.725702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.725904 (+ 202us) server_negotiation.cc:407] Connection header received
0926 07:05:33.726431 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.726433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.726446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.726510 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:33.727460 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.727966 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.728989 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.729201 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.729284 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.731305 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:33.731307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:33.731307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:33.731314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:33.732964 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.733400 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.733402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.733402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.733464 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.733845 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.733845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.733846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.733946 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:33.734000 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.734328 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.734392 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:05:33.734772 20268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.725637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34985 (local address 127.1.217.66:50307)
0926 07:05:33.725782 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.725818 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.726324 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.726583 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.726586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.727144 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:33.727358 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.727363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.728062 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.728063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.728887 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.728889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.729295 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.729296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.729421 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.729873 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:33.729881 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:33.731220 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:33.733053 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.733060 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.733062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.733322 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.733521 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.733522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.733523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.733696 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.734013 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:33.734019 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:33.734193 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.734309 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.734490 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260926 07:05:33.735248 20266 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34985
I20260926 07:05:33.735354 20266 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:33.735561 20266 heartbeater.cc:511] Master 127.1.217.126:34985 requested a full tablet report, sending...
I20260926 07:05:33.735919 19941 ts_manager.cc:194] Registered new tserver with Master: 1d1f5979ba7c4a479ef184e59b8221f0 (127.1.217.66:39415)
I20260926 07:05:33.736297 19941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:50307
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:05:33.759327 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:33.839216 20271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:33.839447 20271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:33.839481 20271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:33.840639 20271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:33.840705 20271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:33.840751 20271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:33.842314 20271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:33.842576 20271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:33.842804 20271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:33.847821 20277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.847833 20279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:33.847921 20276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:33.848400 20271 server_base.cc:1034] running on GCE node
I20260926 07:05:33.848615 20271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:33.848910 20271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:33.850086 20271 hybrid_clock.cc:648] HybridClock initialized: now 1790406333850073 us; error 29 us; skew 500 ppm
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:33.852643 20271 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:05:33.853122 20271 webserver.cc:460] Webserver started at http://127.1.217.67:44009/ using document root <none> and password file <none>
I20260926 07:05:33.853380 20271 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:33.853446 20271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:33.853590 20271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:33.854629 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "5bbcfe3c7a374fdca766be986044487d"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "9a71907885de54a912d4072deb63d22f"
server_key_iv: "60f4f8741bdbad9912681fd164935036"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.854980 20271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "5bbcfe3c7a374fdca766be986044487d"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "9a71907885de54a912d4072deb63d22f"
server_key_iv: "60f4f8741bdbad9912681fd164935036"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.856333 20271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:33.857152 20286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.857304 20271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.857405 20271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "5bbcfe3c7a374fdca766be986044487d"
format_stamp: "Formatted at 2026-09-26 07:05:33 on dist-test-slave-t1mc"
server_key: "9a71907885de54a912d4072deb63d22f"
server_key_iv: "60f4f8741bdbad9912681fd164935036"
server_key_version: "encryptionkey@0"
I20260926 07:05:33.857491 20271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:33.891398 20271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:33.893707 20271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:33.893867 20271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:33.894120 20271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:33.894433 20271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:33.894486 20271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.894527 20271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:33.894552 20271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:33.900332 20271 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:46055
I20260926 07:05:33.900410 20399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:46055 every 8 connection(s)
I20260926 07:05:33.900744 20271 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:33.904191 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20271
I20260926 07:05:33.904285 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:33.904492 1893 external_mini_cluster.cc:1428] Setting key b05bba52aff47e8338fe2d07c149f805
Sep 26 07:05:33 dist-test-slave-t1mc krb5kdc[19896](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406333, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:33.910218 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.901510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:36307 (local address 127.1.217.126:34985)
0926 07:05:33.901572 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.901574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.901777 (+ 203us) server_negotiation.cc:407] Connection header received
0926 07:05:33.902268 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.902269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.902278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.902335 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:33.903263 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.903745 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.905292 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.905538 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.905591 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.907694 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:33.907695 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:33.907695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:33.907701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:33.908938 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.909284 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.909284 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.909285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.909344 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.909621 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.909622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.909622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.909723 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:33.909776 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.910097 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.910157 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:05:33.910439 20402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.901486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34985 (local address 127.1.217.67:36307)
0926 07:05:33.901650 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.901692 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.902188 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.902397 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.902401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.902988 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:33.903184 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.903189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.903842 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.903843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.905215 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.905218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.905599 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.905599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.905711 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.906169 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:33.906177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:33.907609 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:33.909011 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.909017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.909018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.909224 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.909405 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:33.909406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:33.909406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:33.909549 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:33.909767 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:33.909770 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:33.909941 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.910065 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.910184 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260926 07:05:33.910910 20400 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34985
I20260926 07:05:33.910991 20400 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:33.911139 20400 heartbeater.cc:511] Master 127.1.217.126:34985 requested a full tablet report, sending...
I20260926 07:05:33.911465 19941 ts_manager.cc:194] Registered new tserver with Master: 5bbcfe3c7a374fdca766be986044487d (127.1.217.67:46055)
I20260926 07:05:33.911803 19941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:36307
I20260926 07:05:33.916221 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:33.923556 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.917528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53640 (local address 127.1.217.126:34985)
0926 07:05:33.917602 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.917605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.917694 (+ 89us) server_negotiation.cc:407] Connection header received
0926 07:05:33.917778 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.917779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.917789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.917834 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:33.918402 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.918887 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.919603 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.919795 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.919874 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.921223 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:33.921225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:33.921225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:33.921232 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:33.922342 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.922725 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.922727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.922727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.922785 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:33.922996 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:33.922997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:33.922997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:33.923133 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:33.923254 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.923379 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.923444 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:05:33.925005 19941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:33.926469 19941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:33.932942 20411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:46055 (local address 127.0.0.1:44782)
0926 07:05:33.928905 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.928941 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.929027 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.929308 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.929311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.929313 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:33.929387 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.929389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930725 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.930726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.931724 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.931727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932611 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.932687 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.932701 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.932792 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.932856 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":53,"thread_start_us":43,"threads_started":1}
I20260926 07:05:33.932976 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.1.217.67:46055)
0926 07:05:33.929113 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.929118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.929131 (+ 13us) server_negotiation.cc:407] Connection header received
0926 07:05:33.929175 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.929177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.929193 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.929293 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:33.929475 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930598 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.931814 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932503 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932589 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.932652 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.932836 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.932881 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":110,"threads_started":1}
I20260926 07:05:33.933099 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:37047 (local address 127.0.0.1:50374)
0926 07:05:33.929150 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.929158 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.929201 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.929374 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.929375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.929377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:33.929466 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.929468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930740 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.930741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.931941 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.931945 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932866 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.932927 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.932931 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.932963 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.933023 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":404,"threads_started":1}
I20260926 07:05:33.933419 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50374 (local address 127.1.217.65:37047)
0926 07:05:33.929165 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.929168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.929199 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:05:33.929257 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.929259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.929274 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.929551 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:33.929616 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930653 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932029 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932723 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.933195 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.933261 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.933289 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.933344 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260926 07:05:33.933516 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.1.217.66:39415)
0926 07:05:33.929081 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:33.929104 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:33.929374 (+ 270us) server_negotiation.cc:407] Connection header received
0926 07:05:33.929422 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:33.929425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:33.929445 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:33.929577 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:33.929832 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930484 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932062 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932530 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932623 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.932684 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:33.932981 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 07:05:33.933431 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260926 07:05:33.932945 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:33.928681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:39415 (local address 127.0.0.1:53190)
0926 07:05:33.929276 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:33.929284 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:33.929323 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:33.929677 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:33.929678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:33.929680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:33.929718 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.929719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.930585 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.930586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.931956 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:33.931958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:33.932714 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:33.932715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:33.932801 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:33.932804 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:33.932849 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 07:05:33.932883 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":88,"thread_start_us":105,"threads_started":1}
I20260926 07:05:33.933789 20334 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09e6976bf04587a576fe322ff806e6 (DEFAULT_TABLE table=test-table [id=7704e27cf4514b99947efaa3e7ddeacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:33.934154 20334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09e6976bf04587a576fe322ff806e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:33.934144 20066 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09e6976bf04587a576fe322ff806e6 (DEFAULT_TABLE table=test-table [id=7704e27cf4514b99947efaa3e7ddeacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:33.934240 20200 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09e6976bf04587a576fe322ff806e6 (DEFAULT_TABLE table=test-table [id=7704e27cf4514b99947efaa3e7ddeacf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:33.934486 20066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09e6976bf04587a576fe322ff806e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:33.934540 20200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09e6976bf04587a576fe322ff806e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:33.937034 20417 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Bootstrap starting.
I20260926 07:05:33.937644 20418 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Bootstrap starting.
I20260926 07:05:33.937704 20419 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Bootstrap starting.
I20260926 07:05:33.937880 20417 tablet_bootstrap.cc:654] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:33.938290 20419 tablet_bootstrap.cc:654] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:33.938329 20418 tablet_bootstrap.cc:654] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:33.938503 20417 log.cc:824] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:33.938866 20419 log.cc:824] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:33.938884 20418 log.cc:824] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:33.939409 20417 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: No bootstrap required, opened a new log
I20260926 07:05:33.939461 20417 ts_tablet_manager.cc:1397] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:33.939602 20419 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: No bootstrap required, opened a new log
I20260926 07:05:33.939677 20419 ts_tablet_manager.cc:1397] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:33.941131 20417 raft_consensus.cc:348] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.941272 20417 raft_consensus.cc:374] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:33.941295 20417 raft_consensus.cc:729] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56cdd6aa00eb479db5caca843ea31cca, State: Initialized, Role: FOLLOWER
I20260926 07:05:33.941418 20417 consensus_queue.cc:260] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.941309 20419 raft_consensus.cc:348] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.941495 20419 raft_consensus.cc:374] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:33.941543 20419 raft_consensus.cc:729] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1f5979ba7c4a479ef184e59b8221f0, State: Initialized, Role: FOLLOWER
I20260926 07:05:33.941707 20417 ts_tablet_manager.cc:1428] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 07:05:33.941717 20132 heartbeater.cc:503] Master 127.1.217.126:34985 was elected leader, sending a full tablet report...
I20260926 07:05:33.941679 20419 consensus_queue.cc:260] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.942015 20419 ts_tablet_manager.cc:1428] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:05:33.942042 20266 heartbeater.cc:503] Master 127.1.217.126:34985 was elected leader, sending a full tablet report...
I20260926 07:05:33.942281 20418 tablet_bootstrap.cc:492] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: No bootstrap required, opened a new log
I20260926 07:05:33.942335 20418 ts_tablet_manager.cc:1397] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 07:05:33.943836 20418 raft_consensus.cc:348] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.943994 20418 raft_consensus.cc:374] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:33.944039 20418 raft_consensus.cc:729] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbcfe3c7a374fdca766be986044487d, State: Initialized, Role: FOLLOWER
I20260926 07:05:33.944144 20418 consensus_queue.cc:260] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:33.944413 20418 ts_tablet_manager.cc:1428] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:33.944463 20400 heartbeater.cc:503] Master 127.1.217.126:34985 was elected leader, sending a full tablet report...
W20260926 07:05:33.975468 20267 tablet.cc:2369] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:05:34.076824 20133 tablet.cc:2369] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 07:05:34.151445 20401 tablet.cc:2369] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:34.285682 20423 raft_consensus.cc:482] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:34.285809 20423 raft_consensus.cc:504] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:34.286258 20423 leader_election.cc:288] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bbcfe3c7a374fdca766be986044487d (127.1.217.67:46055), 1d1f5979ba7c4a479ef184e59b8221f0 (127.1.217.66:39415)
I20260926 07:05:34.290437 20413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.286528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:57475 (local address 127.1.217.67:46055)
0926 07:05:34.286589 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.286591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.286654 (+ 63us) server_negotiation.cc:407] Connection header received
0926 07:05:34.286712 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.286713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.286722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.286787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:34.287027 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.287780 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.289436 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.290022 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.290093 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.290102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.290314 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.290364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:05:34.290437 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.286561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:53379 (local address 127.1.217.66:39415)
0926 07:05:34.286607 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.286609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.286653 (+ 44us) server_negotiation.cc:407] Connection header received
0926 07:05:34.286712 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.286713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.286722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.286787 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:34.287024 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.287785 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.289437 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.290011 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.290093 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.290102 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.290314 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.290364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:05:34.290462 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.286429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:46055 (local address 127.1.217.65:57475)
0926 07:05:34.286575 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.286587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.286654 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.286806 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.286807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.286811 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:34.286907 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.286910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.287894 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.287896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.289323 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.289327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.290112 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.290113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.290204 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.290213 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.290276 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.290337 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":83,"thread_start_us":62,"threads_started":1}
I20260926 07:05:34.290467 20426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.286440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:39415 (local address 127.1.217.65:53379)
0926 07:05:34.286575 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.286587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.286654 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.286806 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.286807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.286810 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:34.286913 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.286914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.287894 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.287896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.289323 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.289327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.290110 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.290112 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.290204 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.290212 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.290276 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.290337 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":56,"thread_start_us":74,"threads_started":1}
I20260926 07:05:34.290884 20354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09e6976bf04587a576fe322ff806e6" candidate_uuid: "56cdd6aa00eb479db5caca843ea31cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbcfe3c7a374fdca766be986044487d" is_pre_election: true
I20260926 07:05:34.291057 20354 raft_consensus.cc:2454] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cdd6aa00eb479db5caca843ea31cca in term 0.
I20260926 07:05:34.291105 20220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09e6976bf04587a576fe322ff806e6" candidate_uuid: "56cdd6aa00eb479db5caca843ea31cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" is_pre_election: true
I20260926 07:05:34.291260 20220 raft_consensus.cc:2454] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56cdd6aa00eb479db5caca843ea31cca in term 0.
I20260926 07:05:34.291299 20019 leader_election.cc:302] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56cdd6aa00eb479db5caca843ea31cca, 5bbcfe3c7a374fdca766be986044487d; no voters:
I20260926 07:05:34.291436 20423 raft_consensus.cc:2781] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:34.291493 20423 raft_consensus.cc:482] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:34.291517 20423 raft_consensus.cc:3037] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:34.292250 20423 raft_consensus.cc:504] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:34.292392 20423 leader_election.cc:288] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [CANDIDATE]: Term 1 election: Requested vote from peers 5bbcfe3c7a374fdca766be986044487d (127.1.217.67:46055), 1d1f5979ba7c4a479ef184e59b8221f0 (127.1.217.66:39415)
I20260926 07:05:34.292541 20354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09e6976bf04587a576fe322ff806e6" candidate_uuid: "56cdd6aa00eb479db5caca843ea31cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbcfe3c7a374fdca766be986044487d"
I20260926 07:05:34.292613 20354 raft_consensus.cc:3037] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:34.292879 20220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09e6976bf04587a576fe322ff806e6" candidate_uuid: "56cdd6aa00eb479db5caca843ea31cca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d1f5979ba7c4a479ef184e59b8221f0"
I20260926 07:05:34.292972 20220 raft_consensus.cc:3037] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:34.293416 20354 raft_consensus.cc:2454] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cdd6aa00eb479db5caca843ea31cca in term 1.
I20260926 07:05:34.293565 20019 leader_election.cc:302] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56cdd6aa00eb479db5caca843ea31cca, 5bbcfe3c7a374fdca766be986044487d; no voters:
I20260926 07:05:34.293656 20423 raft_consensus.cc:2781] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:34.293797 20423 raft_consensus.cc:686] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [term 1 LEADER]: Becoming Leader. State: Replica: 56cdd6aa00eb479db5caca843ea31cca, State: Running, Role: LEADER
I20260926 07:05:34.293879 20220 raft_consensus.cc:2454] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56cdd6aa00eb479db5caca843ea31cca in term 1.
I20260926 07:05:34.293895 20423 consensus_queue.cc:237] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } }
I20260926 07:05:34.294730 19940 catalog_manager.cc:5495] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca reported cstate change: term changed from 0 to 1, leader changed from <none> to 56cdd6aa00eb479db5caca843ea31cca (127.1.217.65). New cstate: current_term: 1 leader_uuid: "56cdd6aa00eb479db5caca843ea31cca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56cdd6aa00eb479db5caca843ea31cca" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 37047 } health_report { overall_health: HEALTHY } } }
I20260926 07:05:34.373112 20415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.370572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.1.217.65:37047)
0926 07:05:34.370635 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.370637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.370663 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:05:34.370706 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.370707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.370726 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.370797 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:34.370932 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.371450 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.372277 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.372473 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.372561 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.372700 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:34.372854 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:34.372960 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.373015 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.373027 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 07:05:34.374850 20220 raft_consensus.cc:1261] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 1 FOLLOWER]: Refusing update from remote peer 56cdd6aa00eb479db5caca843ea31cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:34.374882 20354 raft_consensus.cc:1261] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 1 FOLLOWER]: Refusing update from remote peer 56cdd6aa00eb479db5caca843ea31cca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:34.375192 20428 consensus_queue.cc:1035] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d1f5979ba7c4a479ef184e59b8221f0" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 39415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:34.375331 20423 consensus_queue.cc:1035] T 4b09e6976bf04587a576fe322ff806e6 P 56cdd6aa00eb479db5caca843ea31cca [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbcfe3c7a374fdca766be986044487d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:34.377358 20431 mvcc.cc:205] Tried to move back new op lower bound from 7333504345596784640 to 7333504345268371456. Current Snapshot: MvccSnapshot[applied={T|T < 7333504345596784640}]
I20260926 07:05:34.377995 20432 mvcc.cc:205] Tried to move back new op lower bound from 7333504345596784640 to 7333504345268371456. Current Snapshot: MvccSnapshot[applied={T|T < 7333504345596784640}]
I20260926 07:05:34.379799 19940 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:34.379894 19940 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:34.380653 19940 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02fac1a0594d43dba2b0fb16a279c65b: Sending DeleteTablet for 3 replicas of tablet 4b09e6976bf04587a576fe322ff806e6
I20260926 07:05:34.381031 20334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09e6976bf04587a576fe322ff806e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:34 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:44782
I20260926 07:05:34.381053 20200 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09e6976bf04587a576fe322ff806e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:34 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:53190
I20260926 07:05:34.381232 20066 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09e6976bf04587a576fe322ff806e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:34 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:50374
I20260926 07:05:34.381309 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20003
I20260926 07:05:34.381906 20441 tablet_replica.cc:330] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: stopping tablet replica
I20260926 07:05:34.382004 20441 raft_consensus.cc:2229] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:34.382109 20441 raft_consensus.cc:2258] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:34.382270 20442 tablet_replica.cc:330] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: stopping tablet replica
I20260926 07:05:34.382357 20442 raft_consensus.cc:2229] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:34.382452 20442 raft_consensus.cc:2258] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:34.382539 20441 ts_tablet_manager.cc:1905] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:34.384099 20441 ts_tablet_manager.cc:1918] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:34.384117 20442 ts_tablet_manager.cc:1905] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:34.384176 20441 log.cc:1196] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/4b09e6976bf04587a576fe322ff806e6
I20260926 07:05:34.384421 20441 ts_tablet_manager.cc:1939] T 4b09e6976bf04587a576fe322ff806e6 P 1d1f5979ba7c4a479ef184e59b8221f0: Deleting consensus metadata
I20260926 07:05:34.384835 19927 catalog_manager.cc:4841] TS 1d1f5979ba7c4a479ef184e59b8221f0 (127.1.217.66:39415): tablet 4b09e6976bf04587a576fe322ff806e6 (table test-table [id=7704e27cf4514b99947efaa3e7ddeacf]) successfully deleted
I20260926 07:05:34.385744 20442 ts_tablet_manager.cc:1918] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:34.385824 20442 log.cc:1196] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/4b09e6976bf04587a576fe322ff806e6
I20260926 07:05:34.386031 20442 ts_tablet_manager.cc:1939] T 4b09e6976bf04587a576fe322ff806e6 P 5bbcfe3c7a374fdca766be986044487d: Deleting consensus metadata
I20260926 07:05:34.386380 19925 catalog_manager.cc:4841] TS 5bbcfe3c7a374fdca766be986044487d (127.1.217.67:46055): tablet 4b09e6976bf04587a576fe322ff806e6 (table test-table [id=7704e27cf4514b99947efaa3e7ddeacf]) successfully deleted
W20260926 07:05:34.387441 19926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:37047 (error 108)
I20260926 07:05:34.387684 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20137
I20260926 07:05:34.388291 20412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.387785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:37047 (local address 127.0.0.1:50402)
0926 07:05:34.388208 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.388252 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:37047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
W20260926 07:05:34.388538 19926 catalog_manager.cc:4584] TS 56cdd6aa00eb479db5caca843ea31cca (127.1.217.65:37047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4b09e6976bf04587a576fe322ff806e6: Network error: Client connection negotiation failed: client connection to 127.1.217.65:37047: connect: Connection refused (error 111)
I20260926 07:05:34.392590 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20271
I20260926 07:05:34.397365 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 19909
2026-09-26T07:05:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2320 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T07:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:34Z Disabled control of system clock
I20260926 07:05:34.424090 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42379
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:34.503772 20448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:34.504002 20448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:34.504031 20448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:34.505364 20448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:34.505405 20448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:34.505419 20448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:34.505457 20448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:34.505483 20448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:34.506798 20448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:34.507047 20448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:34.507332 20448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:34.512254 20454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.512280 20453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.512254 20456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.512619 20448 server_base.cc:1034] running on GCE node
I20260926 07:05:34.512889 20448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:34.513175 20448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:34.514370 20448 hybrid_clock.cc:648] HybridClock initialized: now 1790406334514354 us; error 44 us; skew 500 ppm
I20260926 07:05:34.520704 20448 webserver.cc:460] Webserver started at http://127.1.217.126:36793/ using document root <none> and password file <none>
I20260926 07:05:34.521000 20448 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:34.521061 20448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:34.521229 20448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:34.522348 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "eb0f80604aa54a41a831010723be2cac"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "d9ad49a6d6d83093da3757ee0a050a81"
server_key_iv: "fd8e4c9b2f3c4f347957031aad4eaa87"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.522704 20448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "eb0f80604aa54a41a831010723be2cac"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "d9ad49a6d6d83093da3757ee0a050a81"
server_key_iv: "fd8e4c9b2f3c4f347957031aad4eaa87"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.524078 20448 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:34.524879 20462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.525027 20448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:34.525117 20448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "eb0f80604aa54a41a831010723be2cac"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "d9ad49a6d6d83093da3757ee0a050a81"
server_key_iv: "fd8e4c9b2f3c4f347957031aad4eaa87"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.525187 20448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:34.537986 20448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:34.538261 20448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:34.538363 20448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:34.542337 20448 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42379
I20260926 07:05:34.542368 20514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42379 every 8 connection(s)
I20260926 07:05:34.542712 20448 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:34.543668 20515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:34.545979 20515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: Bootstrap starting.
I20260926 07:05:34.546571 20515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:34.546864 20515 log.cc:824] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:34.547551 20515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: No bootstrap required, opened a new log
I20260926 07:05:34.548255 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20448
I20260926 07:05:34.548385 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:34.548576 1893 external_mini_cluster.cc:1428] Setting key f387638cfcf21ab9f01d7dc4202f20ab
I20260926 07:05:34.549744 20515 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } }
I20260926 07:05:34.549919 20515 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:34.549957 20515 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f80604aa54a41a831010723be2cac, State: Initialized, Role: FOLLOWER
I20260926 07:05:34.550133 20515 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } }
I20260926 07:05:34.550239 20515 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:34.550264 20515 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:34.550333 20515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:34.550856 20515 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } }
I20260926 07:05:34.550940 20515 leader_election.cc:302] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb0f80604aa54a41a831010723be2cac; no voters:
I20260926 07:05:34.551194 20515 leader_election.cc:288] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:34.551236 20520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:34.551357 20520 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [term 1 LEADER]: Becoming Leader. State: Replica: eb0f80604aa54a41a831010723be2cac, State: Running, Role: LEADER
I20260926 07:05:34.551462 20515 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:34.551443 20520 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } }
I20260926 07:05:34.551753 20520 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb0f80604aa54a41a831010723be2cac. Latest consensus state: current_term: 1 leader_uuid: "eb0f80604aa54a41a831010723be2cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } } }
I20260926 07:05:34.551841 20520 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:34.552031 20521 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb0f80604aa54a41a831010723be2cac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb0f80604aa54a41a831010723be2cac" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42379 } } }
I20260926 07:05:34.552138 20521 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:34.552738 20524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:34.553292 20524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:34.554983 20524 catalog_manager.cc:1347] Generated new cluster ID: c7211d855d8a4b6a96509063160f31f1
I20260926 07:05:34.555044 20524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:34.555539 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.549142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.1.217.126:42379)
0926 07:05:34.549374 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.549382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.549420 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:05:34.549704 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.549712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.549790 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.549935 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:34.550180 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.552020 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.553297 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.553884 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.554037 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.554777 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:34.554780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:34.554780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:34.555002 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:34.555043 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:34.555058 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:34.555122 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.555253 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.555334 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260926 07:05:34.575380 20524 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:34.575928 20524 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:34.584407 20524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: Generated new TSK 0
I20260926 07:05:34.584599 20524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:34.587744 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--builtin_ntp_servers=127.1.217.84:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:34.672681 20539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:34.672896 20539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:34.672919 20539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:34.674024 20539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:34.674064 20539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:34.674089 20539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:34.675375 20539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:34.675621 20539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:34.675909 20539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:34.680647 20547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.680673 20544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.680871 20539 server_base.cc:1034] running on GCE node
W20260926 07:05:34.680902 20545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.681156 20539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:34.681416 20539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:34.682605 20539 hybrid_clock.cc:648] HybridClock initialized: now 1790406334682592 us; error 26 us; skew 500 ppm
I20260926 07:05:34.683892 20539 webserver.cc:460] Webserver started at http://127.1.217.65:40819/ using document root <none> and password file <none>
I20260926 07:05:34.684165 20539 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:34.684211 20539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:34.684366 20539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:34.685464 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "d5b90e94aa714f70a96cd657e8ab339b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "bdf74ede721149760638a88b13194b60"
server_key_iv: "3c67bf01283bfc0117b92d1624a5c958"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.685822 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "d5b90e94aa714f70a96cd657e8ab339b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "bdf74ede721149760638a88b13194b60"
server_key_iv: "3c67bf01283bfc0117b92d1624a5c958"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.687084 20539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:34.687822 20553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.688011 20539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:34.688108 20539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "d5b90e94aa714f70a96cd657e8ab339b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "bdf74ede721149760638a88b13194b60"
server_key_iv: "3c67bf01283bfc0117b92d1624a5c958"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.688189 20539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:34.703610 20539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:34.703900 20539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:34.704051 20539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:34.704262 20539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:34.704568 20539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:34.704618 20539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.704653 20539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:34.704679 20539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.710412 20539 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:41069
I20260926 07:05:34.710503 20666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:41069 every 8 connection(s)
I20260926 07:05:34.710808 20539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:05:34.713002 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20539
I20260926 07:05:34.713117 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:34.713330 1893 external_mini_cluster.cc:1428] Setting key 97dd64f4583b635c2c1282a13933614a
I20260926 07:05:34.714233 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--builtin_ntp_servers=127.1.217.84:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:05:34.716391 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.711516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:41347 (local address 127.1.217.126:42379)
0926 07:05:34.711578 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.711580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.711785 (+ 205us) server_negotiation.cc:407] Connection header received
0926 07:05:34.712252 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.712253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.712263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.712313 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:34.713079 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.713717 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.714843 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.715166 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.715236 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.716145 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:34.716151 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:34.716151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:34.716155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:34.716167 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:34.716171 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:34.716215 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.716310 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.716335 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 07:05:34.716691 20669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.711503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42379 (local address 127.1.217.65:41347)
0926 07:05:34.711670 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.711703 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.712173 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.712364 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.712368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.712376 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:34.712971 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.712978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.713802 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.713803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.714759 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.714761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.715255 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.715256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.715399 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.715989 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:34.715989 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:34.716029 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:34.716058 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:34.716230 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:34.716236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:34.716239 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.716309 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.716427 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260926 07:05:34.717249 20667 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42379
I20260926 07:05:34.717339 20667 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:34.717526 20667 heartbeater.cc:511] Master 127.1.217.126:42379 requested a full tablet report, sending...
I20260926 07:05:34.718030 20479 ts_manager.cc:194] Registered new tserver with Master: d5b90e94aa714f70a96cd657e8ab339b (127.1.217.65:41069)
I20260926 07:05:34.718640 20479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.65:41347
W20260926 07:05:34.794723 20670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:34.794939 20670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:34.794976 20670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:34.796123 20670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:34.796180 20670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:34.796223 20670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:34.797590 20670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:34.797833 20670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:34.798121 20670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:34.802894 20678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.802919 20676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.802958 20675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.803227 20670 server_base.cc:1034] running on GCE node
I20260926 07:05:34.803398 20670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:34.803624 20670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:34.804837 20670 hybrid_clock.cc:648] HybridClock initialized: now 1790406334804815 us; error 37 us; skew 500 ppm
I20260926 07:05:34.806028 20670 webserver.cc:460] Webserver started at http://127.1.217.66:45217/ using document root <none> and password file <none>
I20260926 07:05:34.806275 20670 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:34.806324 20670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:34.806463 20670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:34.807528 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "0c652b0c269c4df3836bdd59970fa709"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "8befa2ae2b7e66199f1d9cf1e8ba2d3f"
server_key_iv: "1e020d6fac52ae8ced3446dd4000a386"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.807870 20670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "0c652b0c269c4df3836bdd59970fa709"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "8befa2ae2b7e66199f1d9cf1e8ba2d3f"
server_key_iv: "1e020d6fac52ae8ced3446dd4000a386"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.809209 20670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:34.809929 20684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.810122 20670 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:34.810217 20670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "0c652b0c269c4df3836bdd59970fa709"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "8befa2ae2b7e66199f1d9cf1e8ba2d3f"
server_key_iv: "1e020d6fac52ae8ced3446dd4000a386"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.810290 20670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:34.828537 20670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:34.828869 20670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:34.829016 20670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:34.829231 20670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:34.829555 20670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:34.829605 20670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.829639 20670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:34.829667 20670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.835160 20670 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:42733
I20260926 07:05:34.835204 20797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:42733 every 8 connection(s)
I20260926 07:05:34.835561 20670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:05:34.839810 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20670
I20260926 07:05:34.839917 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:34.840119 1893 external_mini_cluster.cc:1428] Setting key a1c5888401544c33b537b6dbc2900715
I20260926 07:05:34.841370 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--builtin_ntp_servers=127.1.217.84:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:05:34.841535 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.836361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:46097 (local address 127.1.217.126:42379)
0926 07:05:34.836423 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.836425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.836650 (+ 225us) server_negotiation.cc:407] Connection header received
0926 07:05:34.837269 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.837270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.837281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.837341 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:34.838057 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.838550 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.839642 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.839951 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.840096 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.841205 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:34.841206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:34.841207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:34.841210 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:34.841226 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:34.841229 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:34.841279 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.841390 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.841413 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 07:05:34.841812 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.836347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42379 (local address 127.1.217.66:46097)
0926 07:05:34.836529 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.836562 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.837146 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.837449 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.837452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.837461 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:34.837968 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.837972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.838624 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.838625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.839438 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.839440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.840119 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.840120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.840297 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.841035 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:34.841036 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:34.841080 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:34.841109 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:34.841297 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:34.841303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:34.841306 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.841390 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.841525 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 07:05:34.842422 20798 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42379
I20260926 07:05:34.842522 20798 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:34.842770 20798 heartbeater.cc:511] Master 127.1.217.126:42379 requested a full tablet report, sending...
I20260926 07:05:34.843218 20479 ts_manager.cc:194] Registered new tserver with Master: 0c652b0c269c4df3836bdd59970fa709 (127.1.217.66:42733)
I20260926 07:05:34.843629 20479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.66:46097
W20260926 07:05:34.924163 20801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:34.924367 20801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:34.924389 20801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:34.925539 20801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:34.925578 20801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:34.925603 20801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:34.926906 20801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:42379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:34.927153 20801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:34.927459 20801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:34.932447 20809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.932510 20801 server_base.cc:1034] running on GCE node
W20260926 07:05:34.932454 20807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:34.932463 20806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:34.932900 20801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:34.933216 20801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:34.934370 20801 hybrid_clock.cc:648] HybridClock initialized: now 1790406334934351 us; error 39 us; skew 500 ppm
I20260926 07:05:34.935563 20801 webserver.cc:460] Webserver started at http://127.1.217.67:34449/ using document root <none> and password file <none>
I20260926 07:05:34.935802 20801 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:34.935848 20801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:34.935969 20801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:34.937158 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "44d05961925f42d387bddedc7da6108b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "e856553ab01861f0e6cf7a0da2e31136"
server_key_iv: "63072d71c44b50372b68c9a6c5bddfc0"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.937494 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "44d05961925f42d387bddedc7da6108b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "e856553ab01861f0e6cf7a0da2e31136"
server_key_iv: "63072d71c44b50372b68c9a6c5bddfc0"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.938762 20801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:05:34.939517 20815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.939728 20801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:34.939811 20801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "44d05961925f42d387bddedc7da6108b"
format_stamp: "Formatted at 2026-09-26 07:05:34 on dist-test-slave-t1mc"
server_key: "e856553ab01861f0e6cf7a0da2e31136"
server_key_iv: "63072d71c44b50372b68c9a6c5bddfc0"
server_key_version: "encryptionkey@0"
I20260926 07:05:34.939879 20801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:34.946529 20801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:34.946769 20801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:34.946897 20801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:34.947088 20801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:34.947393 20801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:34.947443 20801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.947481 20801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:34.947511 20801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:34.953190 20801 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:37345
I20260926 07:05:34.953229 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:37345 every 8 connection(s)
I20260926 07:05:34.953549 20801 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:34.956492 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20801
I20260926 07:05:34.956609 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:34.956859 1893 external_mini_cluster.cc:1428] Setting key c27c7f109a324bdacce5502788c93b1c
I20260926 07:05:34.958730 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.954205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:40739 (local address 127.1.217.126:42379)
0926 07:05:34.954263 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.954265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.954509 (+ 244us) server_negotiation.cc:407] Connection header received
0926 07:05:34.954985 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.954986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.954997 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.955086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:34.955725 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.956223 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.957508 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.957767 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.957850 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.958540 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:34.958541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:34.958541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:34.958545 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:34.958557 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:34.958561 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:34.958602 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.958649 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.958672 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 07:05:34.958963 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.954214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42379 (local address 127.1.217.67:40739)
0926 07:05:34.954378 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.954431 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.954902 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.955151 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.955155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.955162 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:34.955653 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.955657 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.956293 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.956294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.957395 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.957398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.957862 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.957863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.957970 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.958423 (+ 453us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:34.958424 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:34.958450 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:34.958471 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:34.958602 (+ 131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:34.958604 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:34.958606 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.958652 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.958724 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260926 07:05:34.959416 20929 heartbeater.cc:347] Connected to a master server at 127.1.217.126:42379
I20260926 07:05:34.959503 20929 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:34.959671 20929 heartbeater.cc:511] Master 127.1.217.126:42379 requested a full tablet report, sending...
I20260926 07:05:34.959998 20479 ts_manager.cc:194] Registered new tserver with Master: 44d05961925f42d387bddedc7da6108b (127.1.217.67:37345)
I20260926 07:05:34.960373 20479 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.67:40739
I20260926 07:05:34.968628 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:34.972836 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.970063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.1.217.126:42379)
0926 07:05:34.970147 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.970149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.970182 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:05:34.970232 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.970233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.970242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.970286 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:34.970451 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.970909 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.971629 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.971804 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.971854 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.972540 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:34.972541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:34.972541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:34.972545 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:34.972557 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:34.972561 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:34.972634 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.972694 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.972704 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 07:05:34.974792 20479 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:34.976423 20479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:34.982725 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.978603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37345 (local address 127.0.0.1:38512)
0926 07:05:34.979082 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.979092 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.979136 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.979339 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.979342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.979344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:34.979414 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.979417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980436 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.980437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.981662 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.981664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982427 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.982428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.982496 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.982509 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.982587 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.982638 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":104,"threads_started":1}
I20260926 07:05:34.982816 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.978603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:42733 (local address 127.0.0.1:41622)
0926 07:05:34.978732 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.978782 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.978880 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.979431 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.979432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.979435 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:34.979472 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.979473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980866 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.980867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.981729 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.981731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982574 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.982575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.982630 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.982632 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.982673 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.982704 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 07:05:34.982985 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.978607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41069 (local address 127.0.0.1:48986)
0926 07:05:34.978922 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:34.978933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:34.978998 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:34.979346 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:34.979347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:34.979349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:34.979420 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.979422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980829 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.980830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.981896 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:34.981897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982782 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.982783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:34.982849 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.982853 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:34.982883 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 07:05:34.982922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":101,"threads_started":1}
I20260926 07:05:34.982821 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.979115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.1.217.66:42733)
0926 07:05:34.979285 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.979289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.979299 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:34.979347 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.979348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.979362 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.979437 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:34.979608 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980735 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.981793 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982504 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.982574 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.982662 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.982691 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.982736 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 07:05:34.983268 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.978909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.1.217.67:37345)
0926 07:05:34.979099 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.979104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.979136 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:05:34.979217 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.979220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.979235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.979313 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:34.979543 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980349 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.981755 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982328 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.982423 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.982517 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.983141 (+ 624us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.983191 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 07:05:34.983351 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:34.978920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.1.217.65:41069)
0926 07:05:34.979144 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:34.979148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:34.979158 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:34.979200 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:34.979202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:34.979218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:34.979522 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:34.979581 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.980715 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.981974 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:34.982688 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:34.983097 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:34.983208 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:34.983232 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 07:05:34.983272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260926 07:05:34.983752 20732 tablet_service.cc:1467] Processing CreateTablet for tablet e125a84f35e549aaa86f5ace1170aff6 (DEFAULT_TABLE table=test-table [id=16d7754c04f0481ab3b2029e3f1c0f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:34.983904 20863 tablet_service.cc:1467] Processing CreateTablet for tablet e125a84f35e549aaa86f5ace1170aff6 (DEFAULT_TABLE table=test-table [id=16d7754c04f0481ab3b2029e3f1c0f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:34.983906 20601 tablet_service.cc:1467] Processing CreateTablet for tablet e125a84f35e549aaa86f5ace1170aff6 (DEFAULT_TABLE table=test-table [id=16d7754c04f0481ab3b2029e3f1c0f23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:34.984174 20732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e125a84f35e549aaa86f5ace1170aff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:34.984247 20863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e125a84f35e549aaa86f5ace1170aff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:34.984247 20601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e125a84f35e549aaa86f5ace1170aff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:34.986729 20943 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Bootstrap starting.
I20260926 07:05:34.987463 20945 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Bootstrap starting.
I20260926 07:05:34.987493 20943 tablet_bootstrap.cc:654] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:34.987789 20943 log.cc:824] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:34.988026 20945 tablet_bootstrap.cc:654] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:34.988296 20945 log.cc:824] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:34.988495 20943 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: No bootstrap required, opened a new log
I20260926 07:05:34.988566 20943 ts_tablet_manager.cc:1397] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:34.988606 20944 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: Bootstrap starting.
I20260926 07:05:34.988994 20945 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: No bootstrap required, opened a new log
I20260926 07:05:34.989070 20945 ts_tablet_manager.cc:1397] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:34.989200 20944 tablet_bootstrap.cc:654] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:34.989466 20944 log.cc:824] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:34.990191 20944 tablet_bootstrap.cc:492] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: No bootstrap required, opened a new log
I20260926 07:05:34.990266 20944 ts_tablet_manager.cc:1397] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:34.991231 20943 raft_consensus.cc:348] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.991395 20943 raft_consensus.cc:374] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:34.991429 20943 raft_consensus.cc:729] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c652b0c269c4df3836bdd59970fa709, State: Initialized, Role: FOLLOWER
I20260926 07:05:34.991585 20943 consensus_queue.cc:260] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.991847 20943 ts_tablet_manager.cc:1428] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:05:34.992017 20798 heartbeater.cc:503] Master 127.1.217.126:42379 was elected leader, sending a full tablet report...
I20260926 07:05:34.991916 20945 raft_consensus.cc:348] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.992092 20945 raft_consensus.cc:374] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:34.992127 20945 raft_consensus.cc:729] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d05961925f42d387bddedc7da6108b, State: Initialized, Role: FOLLOWER
I20260926 07:05:34.992231 20945 consensus_queue.cc:260] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.992471 20945 ts_tablet_manager.cc:1428] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:05:34.992494 20929 heartbeater.cc:503] Master 127.1.217.126:42379 was elected leader, sending a full tablet report...
I20260926 07:05:34.992442 20944 raft_consensus.cc:348] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.992585 20944 raft_consensus.cc:374] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:34.992619 20944 raft_consensus.cc:729] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b90e94aa714f70a96cd657e8ab339b, State: Initialized, Role: FOLLOWER
I20260926 07:05:34.992739 20944 consensus_queue.cc:260] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:34.993003 20944 ts_tablet_manager.cc:1428] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:05:34.993091 20667 heartbeater.cc:503] Master 127.1.217.126:42379 was elected leader, sending a full tablet report...
W20260926 07:05:35.086102 20799 tablet.cc:2369] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:35.093120 20949 raft_consensus.cc:482] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:35.093235 20949 raft_consensus.cc:504] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:35.093555 20949 leader_election.cc:288] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d05961925f42d387bddedc7da6108b (127.1.217.67:37345), d5b90e94aa714f70a96cd657e8ab339b (127.1.217.65:41069)
I20260926 07:05:35.096531 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.093754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41069 (local address 127.1.217.66:44735)
0926 07:05:35.093810 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:35.093825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:35.093894 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:35.094081 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:35.094082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:35.094085 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:35.094168 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:35.094170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.094875 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.094876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:35.095709 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:35.095712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.096265 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.096266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:35.096342 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:35.096346 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:35.096403 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 07:05:35.096435 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 07:05:35.096550 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.093858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:44735 (local address 127.1.217.65:41069)
0926 07:05:35.093906 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:35.093908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:35.093923 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:05:35.093980 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:35.093981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:35.093993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:35.094056 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:35.094243 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.094793 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.095801 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.096189 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.096245 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:35.096251 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:35.096419 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 07:05:35.096466 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:05:35.096616 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.093897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:36793 (local address 127.1.217.67:37345)
0926 07:05:35.093940 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:35.093942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:35.094072 (+ 130us) server_negotiation.cc:407] Connection header received
0926 07:05:35.094126 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:35.094128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:35.094139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:35.094204 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:35.094417 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.094917 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.095874 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.096256 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.096329 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:35.096337 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:35.096502 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:05:35.096542 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:05:35.096673 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.093754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37345 (local address 127.1.217.66:36793)
0926 07:05:35.093952 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:35.093962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:35.094063 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:35.094232 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:35.094233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:35.094235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:35.094311 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:35.094312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.095032 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.095033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:35.095787 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:35.095790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.096337 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.096338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:35.096417 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:35.096422 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 07:05:35.096478 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 07:05:35.096513 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":14,"thread_start_us":103,"threads_started":1}
I20260926 07:05:35.096971 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e125a84f35e549aaa86f5ace1170aff6" candidate_uuid: "0c652b0c269c4df3836bdd59970fa709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b90e94aa714f70a96cd657e8ab339b" is_pre_election: true
I20260926 07:05:35.097105 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e125a84f35e549aaa86f5ace1170aff6" candidate_uuid: "0c652b0c269c4df3836bdd59970fa709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d05961925f42d387bddedc7da6108b" is_pre_election: true
I20260926 07:05:35.097144 20621 raft_consensus.cc:2454] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c652b0c269c4df3836bdd59970fa709 in term 0.
I20260926 07:05:35.097218 20883 raft_consensus.cc:2454] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c652b0c269c4df3836bdd59970fa709 in term 0.
I20260926 07:05:35.097399 20685 leader_election.cc:302] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c652b0c269c4df3836bdd59970fa709, d5b90e94aa714f70a96cd657e8ab339b; no voters:
I20260926 07:05:35.097545 20949 raft_consensus.cc:2781] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:35.097601 20949 raft_consensus.cc:482] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:35.097635 20949 raft_consensus.cc:3037] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:35.098315 20949 raft_consensus.cc:504] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:35.098451 20949 leader_election.cc:288] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [CANDIDATE]: Term 1 election: Requested vote from peers 44d05961925f42d387bddedc7da6108b (127.1.217.67:37345), d5b90e94aa714f70a96cd657e8ab339b (127.1.217.65:41069)
I20260926 07:05:35.098593 20883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e125a84f35e549aaa86f5ace1170aff6" candidate_uuid: "0c652b0c269c4df3836bdd59970fa709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d05961925f42d387bddedc7da6108b"
I20260926 07:05:35.098618 20621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e125a84f35e549aaa86f5ace1170aff6" candidate_uuid: "0c652b0c269c4df3836bdd59970fa709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b90e94aa714f70a96cd657e8ab339b"
I20260926 07:05:35.098676 20883 raft_consensus.cc:3037] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:35.098680 20621 raft_consensus.cc:3037] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:35.099560 20883 raft_consensus.cc:2454] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c652b0c269c4df3836bdd59970fa709 in term 1.
I20260926 07:05:35.099560 20621 raft_consensus.cc:2454] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c652b0c269c4df3836bdd59970fa709 in term 1.
I20260926 07:05:35.099745 20688 leader_election.cc:302] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c652b0c269c4df3836bdd59970fa709, 44d05961925f42d387bddedc7da6108b; no voters:
I20260926 07:05:35.099845 20949 raft_consensus.cc:2781] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:35.099992 20949 raft_consensus.cc:686] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 1 LEADER]: Becoming Leader. State: Replica: 0c652b0c269c4df3836bdd59970fa709, State: Running, Role: LEADER
I20260926 07:05:35.100066 20949 consensus_queue.cc:237] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } }
I20260926 07:05:35.100884 20477 catalog_manager.cc:5495] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c652b0c269c4df3836bdd59970fa709 (127.1.217.66). New cstate: current_term: 1 leader_uuid: "0c652b0c269c4df3836bdd59970fa709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c652b0c269c4df3836bdd59970fa709" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 42733 } health_report { overall_health: HEALTHY } } }
I20260926 07:05:35.130553 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.128147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.1.217.66:42733)
0926 07:05:35.128202 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:35.128204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:35.128213 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:35.128255 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:35.128256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:35.128276 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:35.128349 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:35.128527 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.129064 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.129775 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:35.129957 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:35.130003 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:35.130156 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:35.130241 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:35.130314 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:35.130412 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:05:35.130419 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 07:05:35.132447 20883 raft_consensus.cc:1261] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 1 FOLLOWER]: Refusing update from remote peer 0c652b0c269c4df3836bdd59970fa709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:35.132452 20621 raft_consensus.cc:1261] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 1 FOLLOWER]: Refusing update from remote peer 0c652b0c269c4df3836bdd59970fa709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:35.132891 20953 consensus_queue.cc:1035] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b90e94aa714f70a96cd657e8ab339b" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 41069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:35.133052 20949 consensus_queue.cc:1035] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d05961925f42d387bddedc7da6108b" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:35.135732 20956 mvcc.cc:205] Tried to move back new op lower bound from 7333504348699533312 to 7333504348570296320. Current Snapshot: MvccSnapshot[applied={T|T < 7333504348699533312}]
I20260926 07:05:35.136191 20957 mvcc.cc:205] Tried to move back new op lower bound from 7333504348699533312 to 7333504348570296320. Current Snapshot: MvccSnapshot[applied={T|T < 7333504348699533312}]
I20260926 07:05:35.137192 20959 mvcc.cc:205] Tried to move back new op lower bound from 7333504348699533312 to 7333504348570296320. Current Snapshot: MvccSnapshot[applied={T|T < 7333504348699533312}]
I20260926 07:05:35.138192 20478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:35.138283 20478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:35.139137 20478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb0f80604aa54a41a831010723be2cac: Sending DeleteTablet for 3 replicas of tablet e125a84f35e549aaa86f5ace1170aff6
I20260926 07:05:35.139489 20601 tablet_service.cc:1514] Processing DeleteTablet for tablet e125a84f35e549aaa86f5ace1170aff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:35 UTC) from {username='slave'} at 127.0.0.1:48986
I20260926 07:05:35.139501 20863 tablet_service.cc:1514] Processing DeleteTablet for tablet e125a84f35e549aaa86f5ace1170aff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:35 UTC) from {username='slave'} at 127.0.0.1:38512
I20260926 07:05:35.139743 20966 tablet_replica.cc:330] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b: stopping tablet replica
I20260926 07:05:35.139739 20732 tablet_service.cc:1514] Processing DeleteTablet for tablet e125a84f35e549aaa86f5ace1170aff6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:35 UTC) from {username='slave'} at 127.0.0.1:41622
I20260926 07:05:35.139904 20966 raft_consensus.cc:2229] T e125a84f35e549aaa86f5ace1170aff6 P d5b90e94aa714f70a96cd657e8ab339b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:35.139942 20967 tablet_replica.cc:330] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709: stopping tablet replica
I20260926 07:05:35.140048 20967 raft_consensus.cc:2229] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 1 LEADER]: Raft consensus shutting down.
I20260926 07:05:35.140125 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20539
I20260926 07:05:35.140146 20967 raft_consensus.cc:2258] T e125a84f35e549aaa86f5ace1170aff6 P 0c652b0c269c4df3836bdd59970fa709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:35.141734 20965 tablet_replica.cc:330] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: stopping tablet replica
I20260926 07:05:35.141855 20965 raft_consensus.cc:2229] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:35.141966 20965 raft_consensus.cc:2258] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:35.142473 20965 ts_tablet_manager.cc:1905] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:35.144287 20965 ts_tablet_manager.cc:1918] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:35.144402 20965 log.cc:1196] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/e125a84f35e549aaa86f5ace1170aff6
I20260926 07:05:35.144675 20965 ts_tablet_manager.cc:1939] T e125a84f35e549aaa86f5ace1170aff6 P 44d05961925f42d387bddedc7da6108b: Deleting consensus metadata
I20260926 07:05:35.145123 20466 catalog_manager.cc:4841] TS 44d05961925f42d387bddedc7da6108b (127.1.217.67:37345): tablet e125a84f35e549aaa86f5ace1170aff6 (table test-table [id=16d7754c04f0481ab3b2029e3f1c0f23]) successfully deleted
W20260926 07:05:35.147030 20463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:41069 (error 108)
I20260926 07:05:35.147511 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.147384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:41069 (local address 127.0.0.1:48988)
0926 07:05:35.147438 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:35.147477 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 07:05:35.147620 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20670
W20260926 07:05:35.151254 20463 catalog_manager.cc:4584] TS d5b90e94aa714f70a96cd657e8ab339b (127.1.217.65:41069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e125a84f35e549aaa86f5ace1170aff6: Network error: Client connection negotiation failed: client connection to 127.1.217.65:41069: connect: Connection refused (error 111)
I20260926 07:05:35.153182 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:35.153066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:42733 (local address 127.0.0.1:41638)
0926 07:05:35.153120 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:35.153146 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.66:42733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 07:05:35.153170 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20801
I20260926 07:05:35.158576 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20448
2026-09-26T07:05:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (760 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:35 dist-test-slave-t1mc krb5kdc[20971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:35 dist-test-slave-t1mc krb5kdc[20971](info): set up 2 sockets
Sep 26 07:05:35 dist-test-slave-t1mc krb5kdc[20971](info): commencing operation
krb5kdc: starting...
W20260926 07:05:36.192629 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:36.319705 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34551
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:36.399288 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:36.399518 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:36.399555 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:36.400978 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:36.401038 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:36.401065 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:36.401095 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:36.401122 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:36.402634 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:36.402918 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:36.403146 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:36.407995 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.407938 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.408192 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:36.408329 20984 server_base.cc:1034] running on GCE node
I20260926 07:05:36.408667 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:36.408994 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:36.410200 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790406336410183 us; error 23 us; skew 500 ppm
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:36.412690 20984 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:36.413225 20984 webserver.cc:460] Webserver started at http://127.1.217.126:40059/ using document root <none> and password file <none>
I20260926 07:05:36.413455 20984 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:36.413519 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:36.413676 20984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:36.414709 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "085375705c57440fa1065ec5551edbce"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "ef52939d743f9164be4b881dca03285a"
server_key_iv: "b5ae6a718f2c13e6d24bad06e82c5f1b"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.415053 20984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "085375705c57440fa1065ec5551edbce"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "ef52939d743f9164be4b881dca03285a"
server_key_iv: "b5ae6a718f2c13e6d24bad06e82c5f1b"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.416348 20984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:36.417078 20999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.417243 20984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:36.417330 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "085375705c57440fa1065ec5551edbce"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "ef52939d743f9164be4b881dca03285a"
server_key_iv: "b5ae6a718f2c13e6d24bad06e82c5f1b"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.417407 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:36.429484 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:36.431406 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:36.431571 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:36.435335 20984 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34551
I20260926 07:05:36.435375 21051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34551 every 8 connection(s)
I20260926 07:05:36.435719 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:36.436547 21052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:36.438769 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: Bootstrap starting.
I20260926 07:05:36.439306 21052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:36.439533 21052 log.cc:824] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:36.440171 21052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: No bootstrap required, opened a new log
I20260926 07:05:36.441774 21052 raft_consensus.cc:348] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } }
I20260926 07:05:36.441910 21052 raft_consensus.cc:374] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:36.441931 21052 raft_consensus.cc:729] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 085375705c57440fa1065ec5551edbce, State: Initialized, Role: FOLLOWER
I20260926 07:05:36.442060 21052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } }
I20260926 07:05:36.442139 21052 raft_consensus.cc:388] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:36.442162 21052 raft_consensus.cc:482] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:36.442191 21052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:36.442664 21052 raft_consensus.cc:504] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } }
I20260926 07:05:36.442754 21052 leader_election.cc:302] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 085375705c57440fa1065ec5551edbce; no voters:
I20260926 07:05:36.442873 21052 leader_election.cc:288] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:36.442947 21055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:36.443099 21055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [term 1 LEADER]: Becoming Leader. State: Replica: 085375705c57440fa1065ec5551edbce, State: Running, Role: LEADER
I20260926 07:05:36.443147 21052 sys_catalog.cc:564] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:36.443186 21055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } }
I20260926 07:05:36.443567 21057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 085375705c57440fa1065ec5551edbce. Latest consensus state: current_term: 1 leader_uuid: "085375705c57440fa1065ec5551edbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } } }
I20260926 07:05:36.443667 21057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:36.443696 21056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "085375705c57440fa1065ec5551edbce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "085375705c57440fa1065ec5551edbce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34551 } } }
I20260926 07:05:36.443746 21056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:36.444314 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 20984
I20260926 07:05:36.444442 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:36.444429 21063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:36.444687 1893 external_mini_cluster.cc:1428] Setting key c578b9b75e15bb4e9461a237e0290270
I20260926 07:05:36.445016 21063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:36.446813 21063 catalog_manager.cc:1347] Generated new cluster ID: 8a1e65d05f9d4409bbbf3ab0d4c0b0c2
I20260926 07:05:36.446875 21063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:36.451931 21063 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:36.452410 21063 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:36.456830 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.445218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.1.217.126:34551)
0926 07:05:36.445380 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.445390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.445523 (+ 133us) server_negotiation.cc:407] Connection header received
0926 07:05:36.445840 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.445849 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.446208 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.446361 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:36.447112 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.448464 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.449588 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.450209 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.450308 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.453227 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:36.453231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:36.453231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:36.453243 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:36.455200 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.455613 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.455616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.455616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.455716 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.455927 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.455928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.455928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.456179 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:36.456332 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.456431 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.456568 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260926 07:05:36.462199 21063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: Generated new TSK 0
I20260926 07:05:36.462369 21063 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.1.217.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
Entry for principal kudu/127.1.217.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab.
I20260926 07:05:36.491767 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:36.572930 21078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:36.573134 21078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:36.573158 21078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:36.574333 21078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:36.574375 21078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:36.574427 21078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:36.576000 21078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.65.keytab
--principal=kudu/127.1.217.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:36.576267 21078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:36.576508 21078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:36.581420 21084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.581427 21086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.581427 21083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:36.581668 21078 server_base.cc:1034] running on GCE node
I20260926 07:05:36.581887 21078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:36.582127 21078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:36.583328 21078 hybrid_clock.cc:648] HybridClock initialized: now 1790406336583318 us; error 40 us; skew 500 ppm
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:36.586074 21078 init.cc:377] Logged in from keytab as kudu/127.1.217.65@KRBTEST.COM (short username kudu)
I20260926 07:05:36.586508 21078 webserver.cc:460] Webserver started at http://127.1.217.65:43033/ using document root <none> and password file <none>
I20260926 07:05:36.586758 21078 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:36.586828 21078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:36.586973 21078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:36.588042 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "a386f1907d01464e9e647e6312bbe314"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "b28ceb0165fb482cf81f441c2cc6afdc"
server_key_iv: "d3a2b65b43e34863abc5abc4efae2223"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.588400 21078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "a386f1907d01464e9e647e6312bbe314"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "b28ceb0165fb482cf81f441c2cc6afdc"
server_key_iv: "d3a2b65b43e34863abc5abc4efae2223"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.589823 21078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:36.590582 21093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.590768 21078 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:05:36.590870 21078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "a386f1907d01464e9e647e6312bbe314"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "b28ceb0165fb482cf81f441c2cc6afdc"
server_key_iv: "d3a2b65b43e34863abc5abc4efae2223"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.590942 21078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:36.609508 21078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:36.611862 21078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:36.612039 21078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:36.612376 21078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:36.612867 21078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:36.612936 21078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.612980 21078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:36.613009 21078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.618817 21078 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:44283
I20260926 07:05:36.618907 21206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:44283 every 8 connection(s)
I20260926 07:05:36.619225 21078 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.65@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:36.626720 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21078
I20260926 07:05:36.626809 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:36.627034 1893 external_mini_cluster.cc:1428] Setting key 98a6c12b4fd16206d2356e3606ec85f6
I20260926 07:05:36.629552 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.620046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:48535 (local address 127.1.217.126:34551)
0926 07:05:36.620090 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.620093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.620304 (+ 211us) server_negotiation.cc:407] Connection header received
0926 07:05:36.620960 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.620962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.620974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.621026 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:36.622039 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.622555 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.623577 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.623794 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.623867 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.626091 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:36.626092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:36.626093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:36.626100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:36.627825 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.628343 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.628344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.628345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.628401 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.628754 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.628755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.628755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.628924 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:36.628987 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.629323 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.629431 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:05:36.629681 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.619990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34551 (local address 127.1.217.65:48535)
0926 07:05:36.620179 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.620215 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.620848 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.621158 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.621162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.621736 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:36.621934 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.621938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.622661 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.622662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.623473 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.623475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.623886 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.623887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.624021 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.624525 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:36.624533 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:36.625970 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:36.627928 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.627936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.627938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.628263 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.628470 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.628471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.628471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.628618 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.628992 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:36.628998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:36.629164 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.629287 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.629481 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260926 07:05:36.630157 21207 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34551
I20260926 07:05:36.630251 21207 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:36.630420 21207 heartbeater.cc:511] Master 127.1.217.126:34551 requested a full tablet report, sending...
I20260926 07:05:36.630970 21016 ts_manager.cc:194] Registered new tserver with Master: a386f1907d01464e9e647e6312bbe314 (127.1.217.65:44283)
I20260926 07:05:36.631902 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.65@KRBTEST.COM'} at 127.1.217.65:48535
WARNING: no policy specified for kudu/127.1.217.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
Entry for principal kudu/127.1.217.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab.
I20260926 07:05:36.652098 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:36.732568 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:36.732797 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:36.732821 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:36.734009 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:36.734050 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:36.734110 21212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:36.735515 21212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.66.keytab
--principal=kudu/127.1.217.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:36.735790 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:36.736014 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:36.740865 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.740855 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.740855 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:36.741092 21212 server_base.cc:1034] running on GCE node
I20260926 07:05:36.741257 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:36.741441 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:36.742596 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790406336742574 us; error 44 us; skew 500 ppm
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:36.745397 21212 init.cc:377] Logged in from keytab as kudu/127.1.217.66@KRBTEST.COM (short username kudu)
I20260926 07:05:36.745826 21212 webserver.cc:460] Webserver started at http://127.1.217.66:44813/ using document root <none> and password file <none>
I20260926 07:05:36.746030 21212 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:36.746097 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:36.746204 21212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:36.747188 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "b4ec577e9a5a452b8171d68b19a2e4d1"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "79e4357ad47f087971299be4a33ae40c"
server_key_iv: "3bb454506a041618ac1c17e5ce3705fb"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.747488 21212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "b4ec577e9a5a452b8171d68b19a2e4d1"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "79e4357ad47f087971299be4a33ae40c"
server_key_iv: "3bb454506a041618ac1c17e5ce3705fb"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.748718 21212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:36.749457 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.749630 21212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:36.749727 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "b4ec577e9a5a452b8171d68b19a2e4d1"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "79e4357ad47f087971299be4a33ae40c"
server_key_iv: "3bb454506a041618ac1c17e5ce3705fb"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.749804 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:36.761865 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:36.764209 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:36.764372 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:36.764614 21212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:36.764966 21212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:36.765020 21212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.765053 21212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:36.765079 21212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.770870 21212 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:43803
I20260926 07:05:36.770921 21340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:43803 every 8 connection(s)
I20260926 07:05:36.771225 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:05:36.776660 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21212
I20260926 07:05:36.776809 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:36.777012 1893 external_mini_cluster.cc:1428] Setting key 53ce1f50fe5522535b03b1ce8910ce26
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.66@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:36.780706 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.771930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:59759 (local address 127.1.217.126:34551)
0926 07:05:36.771979 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.771981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.772154 (+ 173us) server_negotiation.cc:407] Connection header received
0926 07:05:36.772645 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.772646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.772655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.772723 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:36.773656 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.774142 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.775104 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.775297 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.775346 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.777779 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:36.777780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:36.777780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:36.777788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:36.779437 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.779817 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.779818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.779819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.779885 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.780105 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.780106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.780106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.780250 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:36.780318 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.780555 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.780644 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 07:05:36.780843 21343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.771886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34551 (local address 127.1.217.66:59759)
0926 07:05:36.772038 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.772080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.772560 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.772829 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.772833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.773376 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:36.773580 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.773585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.774239 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.774239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.775030 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.775032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.775368 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.775369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.775470 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.775993 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:36.776002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:36.777678 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:36.779520 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.779526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.779527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.779744 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.779950 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.779950 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.779951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.780044 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.780314 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:36.780318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:36.780427 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.780508 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.780625 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260926 07:05:36.781333 21341 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34551
I20260926 07:05:36.781433 21341 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:36.781613 21341 heartbeater.cc:511] Master 127.1.217.126:34551 requested a full tablet report, sending...
I20260926 07:05:36.781967 21016 ts_manager.cc:194] Registered new tserver with Master: b4ec577e9a5a452b8171d68b19a2e4d1 (127.1.217.66:43803)
I20260926 07:05:36.782409 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.66@KRBTEST.COM'} at 127.1.217.66:59759
WARNING: no policy specified for kudu/127.1.217.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
Entry for principal kudu/127.1.217.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab.
I20260926 07:05:36.803114 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:38055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:36.885428 21346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:36.885634 21346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:36.885658 21346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:36.886749 21346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:36.886785 21346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:36.886809 21346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:36.888156 21346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:38055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.67.keytab
--principal=kudu/127.1.217.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:34551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:36.888331 21346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:36.888497 21346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:36.893281 21354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.893287 21352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:36.893330 21351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:36.893543 21346 server_base.cc:1034] running on GCE node
I20260926 07:05:36.893733 21346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:36.893996 21346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:36.895175 21346 hybrid_clock.cc:648] HybridClock initialized: now 1790406336895163 us; error 41 us; skew 500 ppm
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:36.897915 21346 init.cc:377] Logged in from keytab as kudu/127.1.217.67@KRBTEST.COM (short username kudu)
I20260926 07:05:36.898362 21346 webserver.cc:460] Webserver started at http://127.1.217.67:43863/ using document root <none> and password file <none>
I20260926 07:05:36.898622 21346 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:36.898691 21346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:36.898834 21346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:36.900153 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "87af9bf1631110e1639a2bae0fdc68cb"
server_key_iv: "d6fb2be8adc6de329308878c86c42cde"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.900537 21346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "87af9bf1631110e1639a2bae0fdc68cb"
server_key_iv: "d6fb2be8adc6de329308878c86c42cde"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.901922 21346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 07:05:36.902673 21361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.902845 21346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:36.902932 21346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d"
format_stamp: "Formatted at 2026-09-26 07:05:36 on dist-test-slave-t1mc"
server_key: "87af9bf1631110e1639a2bae0fdc68cb"
server_key_iv: "d6fb2be8adc6de329308878c86c42cde"
server_key_version: "encryptionkey@0"
I20260926 07:05:36.903018 21346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:36.914471 21346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:36.916360 21346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:36.916522 21346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:36.916754 21346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:36.917202 21346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:36.917258 21346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.917299 21346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:36.917327 21346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:36.923394 21346 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:37229
I20260926 07:05:36.923467 21474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:37229 every 8 connection(s)
I20260926 07:05:36.923800 21346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:36.928249 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21346
I20260926 07:05:36.928337 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:36.928568 1893 external_mini_cluster.cc:1428] Setting key ad85b1db493b3acb49b0018425f642e1
Sep 26 07:05:36 dist-test-slave-t1mc krb5kdc[20971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406336, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.67@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:36.933073 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.924585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:45163 (local address 127.1.217.126:34551)
0926 07:05:36.924634 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.924636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.924857 (+ 221us) server_negotiation.cc:407] Connection header received
0926 07:05:36.925346 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.925347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.925356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.925410 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:36.926345 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.926826 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.927843 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.928029 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.928165 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.930569 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:36.930570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:36.930571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:36.930577 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:36.931797 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.932177 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.932178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.932179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.932231 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.932451 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.932452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.932453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.932598 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:36.932654 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.932940 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.933004 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:05:36.933284 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.924528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:34551 (local address 127.1.217.67:45163)
0926 07:05:36.924709 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.924752 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.925261 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.925477 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.925480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.926045 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:36.926266 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.926271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.926918 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.926919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.927757 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.927759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.928172 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.928174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.928335 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.929009 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:05:36.929018 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:36.930470 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:36.931876 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.931882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.931884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.932110 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.932291 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:05:36.932292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:05:36.932292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:05:36.932391 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:05:36.932660 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:36.932664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:36.932834 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.932921 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.933046 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260926 07:05:36.933735 21475 heartbeater.cc:347] Connected to a master server at 127.1.217.126:34551
I20260926 07:05:36.933830 21475 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:36.933982 21475 heartbeater.cc:511] Master 127.1.217.126:34551 requested a full tablet report, sending...
I20260926 07:05:36.934329 21016 ts_manager.cc:194] Registered new tserver with Master: fab2c2465da14cd78ca40aa2bfa1ab7d (127.1.217.67:37229)
I20260926 07:05:36.934711 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.1.217.67@KRBTEST.COM'} at 127.1.217.67:45163
I20260926 07:05:36.940210 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:36.947922 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.941699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.1.217.126:34551)
0926 07:05:36.941765 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.941767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.941845 (+ 78us) server_negotiation.cc:407] Connection header received
0926 07:05:36.941882 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.941882 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.941890 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.941946 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:36.942527 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.943004 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.943939 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.944149 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.944224 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.945520 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:36.945521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:36.945522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:36.945528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:36.946903 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.947241 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.947242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.947242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.947300 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:36.947500 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:36.947500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:36.947501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:36.947633 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:36.947706 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.947799 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.947859 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:05:36.949574 21016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:36.951130 21016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:36.957280 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37229 (local address 127.0.0.1:35634)
0926 07:05:36.953747 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.953757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.953799 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.954089 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.954092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.954094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:36.954168 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.954171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.955056 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.955057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.956127 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.956130 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.956980 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.956981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.957060 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.957074 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.957142 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.957192 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260926 07:05:36.957525 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35634 (local address 127.1.217.67:37229)
0926 07:05:36.953904 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.953909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.953919 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:36.953963 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.953965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.953977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.954077 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:36.954275 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.954966 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.956230 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.956754 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.956885 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.956955 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.957406 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.957453 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260926 07:05:36.958304 21483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:44283 (local address 127.0.0.1:51292)
0926 07:05:36.953428 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.953471 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.953537 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.954467 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.954468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.954470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:36.954510 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.954512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.955512 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.955513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.956631 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.956633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.958017 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.958018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.958091 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.958095 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.958158 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.958206 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":6,"thread_start_us":80,"threads_started":1}
I20260926 07:05:36.958304 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43803 (local address 127.0.0.1:52252)
0926 07:05:36.953666 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:36.953677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:36.953732 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:36.954109 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:36.954109 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:36.954111 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:36.954168 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.954170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.956062 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.956063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.957059 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:36.957061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.958019 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.958019 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:36.958101 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.958104 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:36.958158 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 07:05:36.958205 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260926 07:05:36.958307 21409 tablet_service.cc:1467] Processing CreateTablet for tablet 27cacfdbe1c9487eb58856692ea19714 (DEFAULT_TABLE table=test-table [id=3749fe7ad9404e758d5c8f28ed82503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:36.958684 21409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27cacfdbe1c9487eb58856692ea19714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:36.958813 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.1.217.65:44283)
0926 07:05:36.954316 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.954320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.954328 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:05:36.954371 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.954372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.954383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.954470 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:36.954567 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.955398 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.956927 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.957928 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.958004 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.958056 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.958181 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.958721 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":83,"threads_started":1}
I20260926 07:05:36.959209 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:36.953477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.1.217.66:43803)
0926 07:05:36.953867 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:36.953872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:36.953881 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:36.953925 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:36.953927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:36.953960 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:36.954032 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:36.955064 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.955941 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.957152 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:36.957842 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:36.957909 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:36.957966 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:36.959072 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0926 07:05:36.959121 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":92,"threads_started":1}
I20260926 07:05:36.959432 21141 tablet_service.cc:1467] Processing CreateTablet for tablet 27cacfdbe1c9487eb58856692ea19714 (DEFAULT_TABLE table=test-table [id=3749fe7ad9404e758d5c8f28ed82503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:36.959779 21141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27cacfdbe1c9487eb58856692ea19714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:36.959849 21275 tablet_service.cc:1467] Processing CreateTablet for tablet 27cacfdbe1c9487eb58856692ea19714 (DEFAULT_TABLE table=test-table [id=3749fe7ad9404e758d5c8f28ed82503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:36.960153 21275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27cacfdbe1c9487eb58856692ea19714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:36.961783 21489 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Bootstrap starting.
I20260926 07:05:36.962414 21489 tablet_bootstrap.cc:654] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:36.962929 21489 log.cc:824] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:36.963531 21490 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: Bootstrap starting.
I20260926 07:05:36.963680 21489 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: No bootstrap required, opened a new log
I20260926 07:05:36.963773 21489 ts_tablet_manager.cc:1397] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:36.964156 21490 tablet_bootstrap.cc:654] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:36.964620 21490 log.cc:824] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:36.965013 21492 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Bootstrap starting.
I20260926 07:05:36.965445 21490 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: No bootstrap required, opened a new log
I20260926 07:05:36.965516 21490 ts_tablet_manager.cc:1397] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 07:05:36.965468 21489 raft_consensus.cc:348] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.965672 21492 tablet_bootstrap.cc:654] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:36.965682 21489 raft_consensus.cc:374] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:36.965721 21489 raft_consensus.cc:729] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab2c2465da14cd78ca40aa2bfa1ab7d, State: Initialized, Role: FOLLOWER
I20260926 07:05:36.965842 21489 consensus_queue.cc:260] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.966114 21489 ts_tablet_manager.cc:1428] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 07:05:36.966209 21492 log.cc:824] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:36.966241 21475 heartbeater.cc:503] Master 127.1.217.126:34551 was elected leader, sending a full tablet report...
I20260926 07:05:36.966954 21492 tablet_bootstrap.cc:492] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: No bootstrap required, opened a new log
I20260926 07:05:36.967073 21492 ts_tablet_manager.cc:1397] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 07:05:36.967617 21490 raft_consensus.cc:348] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.967767 21490 raft_consensus.cc:374] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:36.967813 21490 raft_consensus.cc:729] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a386f1907d01464e9e647e6312bbe314, State: Initialized, Role: FOLLOWER
I20260926 07:05:36.967921 21490 consensus_queue.cc:260] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.968210 21490 ts_tablet_manager.cc:1428] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 07:05:36.968225 21207 heartbeater.cc:503] Master 127.1.217.126:34551 was elected leader, sending a full tablet report...
I20260926 07:05:36.969054 21492 raft_consensus.cc:348] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.969249 21492 raft_consensus.cc:374] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:36.969309 21492 raft_consensus.cc:729] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ec577e9a5a452b8171d68b19a2e4d1, State: Initialized, Role: FOLLOWER
I20260926 07:05:36.969415 21492 consensus_queue.cc:260] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:36.969671 21492 ts_tablet_manager.cc:1428] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 07:05:36.969705 21341 heartbeater.cc:503] Master 127.1.217.126:34551 was elected leader, sending a full tablet report...
I20260926 07:05:37.011665 21496 raft_consensus.cc:482] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:37.011852 21496 raft_consensus.cc:504] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:37.012296 21496 leader_election.cc:288] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fab2c2465da14cd78ca40aa2bfa1ab7d (127.1.217.67:37229), b4ec577e9a5a452b8171d68b19a2e4d1 (127.1.217.66:43803)
I20260926 07:05:37.015569 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.012742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:46819 (local address 127.1.217.66:43803)
0926 07:05:37.012808 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.012811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.012871 (+ 60us) server_negotiation.cc:407] Connection header received
0926 07:05:37.012934 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.012935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.012945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.013000 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.013162 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.013621 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.014810 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.015177 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.015280 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.015288 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.015453 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.015494 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:05:37.015575 21488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.012734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:41107 (local address 127.1.217.67:37229)
0926 07:05:37.012799 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.012801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.012810 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:37.012840 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.012841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.012857 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.012935 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.013095 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.013574 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.014811 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.015189 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.015279 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.015288 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.015466 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.015505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 07:05:37.015587 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.012476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:37229 (local address 127.1.217.65:41107)
0926 07:05:37.012535 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.012546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.012623 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.012940 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.012941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.012943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.013018 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.013020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.013671 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.013672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.014712 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.014714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.015276 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.015277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.015379 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.015382 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.015446 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.015502 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":6}
I20260926 07:05:37.015873 21429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27cacfdbe1c9487eb58856692ea19714" candidate_uuid: "a386f1907d01464e9e647e6312bbe314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" is_pre_election: true
I20260926 07:05:37.015599 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.012575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:43803 (local address 127.1.217.65:46819)
0926 07:05:37.012820 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.012830 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.012883 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.013006 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.013007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.013008 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.013078 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.013079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.013703 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.013704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.014721 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.014724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.015270 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.015271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.015341 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.015345 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.015446 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.015495 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":6,"thread_start_us":144,"threads_started":1}
I20260926 07:05:37.016021 21429 raft_consensus.cc:2454] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a386f1907d01464e9e647e6312bbe314 in term 0.
I20260926 07:05:37.016299 21096 leader_election.cc:302] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a386f1907d01464e9e647e6312bbe314, fab2c2465da14cd78ca40aa2bfa1ab7d; no voters:
I20260926 07:05:37.016417 21496 raft_consensus.cc:2781] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:37.016418 21295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27cacfdbe1c9487eb58856692ea19714" candidate_uuid: "a386f1907d01464e9e647e6312bbe314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" is_pre_election: true
I20260926 07:05:37.016471 21496 raft_consensus.cc:482] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:37.016526 21496 raft_consensus.cc:3037] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.016567 21295 raft_consensus.cc:2454] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a386f1907d01464e9e647e6312bbe314 in term 0.
I20260926 07:05:37.017310 21496 raft_consensus.cc:504] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:37.017463 21496 leader_election.cc:288] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [CANDIDATE]: Term 1 election: Requested vote from peers fab2c2465da14cd78ca40aa2bfa1ab7d (127.1.217.67:37229), b4ec577e9a5a452b8171d68b19a2e4d1 (127.1.217.66:43803)
I20260926 07:05:37.017639 21429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27cacfdbe1c9487eb58856692ea19714" candidate_uuid: "a386f1907d01464e9e647e6312bbe314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d"
I20260926 07:05:37.017707 21429 raft_consensus.cc:3037] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.018359 21429 raft_consensus.cc:2454] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a386f1907d01464e9e647e6312bbe314 in term 1.
I20260926 07:05:37.018553 21096 leader_election.cc:302] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a386f1907d01464e9e647e6312bbe314, fab2c2465da14cd78ca40aa2bfa1ab7d; no voters:
I20260926 07:05:37.018576 21295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27cacfdbe1c9487eb58856692ea19714" candidate_uuid: "a386f1907d01464e9e647e6312bbe314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1"
I20260926 07:05:37.018661 21496 raft_consensus.cc:2781] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:37.018692 21295 raft_consensus.cc:3037] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.018836 21496 raft_consensus.cc:686] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [term 1 LEADER]: Becoming Leader. State: Replica: a386f1907d01464e9e647e6312bbe314, State: Running, Role: LEADER
I20260926 07:05:37.018950 21496 consensus_queue.cc:237] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } }
I20260926 07:05:37.019683 21295 raft_consensus.cc:2454] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a386f1907d01464e9e647e6312bbe314 in term 1.
I20260926 07:05:37.019729 21016 catalog_manager.cc:5495] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 reported cstate change: term changed from 0 to 1, leader changed from <none> to a386f1907d01464e9e647e6312bbe314 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "a386f1907d01464e9e647e6312bbe314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a386f1907d01464e9e647e6312bbe314" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 44283 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 } health_report { overall_health: UNKNOWN } } }
W20260926 07:05:37.021741 21342 tablet.cc:2369] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:37.033185 21486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.030982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51306 (local address 127.1.217.65:44283)
0926 07:05:37.031034 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.031036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.031052 (+ 16us) server_negotiation.cc:407] Connection header received
0926 07:05:37.031098 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.031099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.031115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.031176 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:37.031303 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.031774 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.032470 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.032653 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.032704 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.032876 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:37.032971 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:37.033032 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.033091 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.033108 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:05:37.034933 21429 raft_consensus.cc:1261] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 1 FOLLOWER]: Refusing update from remote peer a386f1907d01464e9e647e6312bbe314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:37.034937 21295 raft_consensus.cc:1261] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 1 FOLLOWER]: Refusing update from remote peer a386f1907d01464e9e647e6312bbe314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:37.035332 21499 consensus_queue.cc:1035] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4ec577e9a5a452b8171d68b19a2e4d1" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 43803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:37.035473 21496 consensus_queue.cc:1035] T 27cacfdbe1c9487eb58856692ea19714 P a386f1907d01464e9e647e6312bbe314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fab2c2465da14cd78ca40aa2bfa1ab7d" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 37229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:37.039552 21505 mvcc.cc:205] Tried to move back new op lower bound from 7333504356492374016 to 7333504356430102528. Current Snapshot: MvccSnapshot[applied={T|T < 7333504356492374016}]
I20260926 07:05:37.039649 21503 mvcc.cc:205] Tried to move back new op lower bound from 7333504356492374016 to 7333504356430102528. Current Snapshot: MvccSnapshot[applied={T|T < 7333504356492374016 or (T in {7333504356492374016})}]
I20260926 07:05:37.039772 21015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:37.039860 21015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:37.040588 21015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 085375705c57440fa1065ec5551edbce: Sending DeleteTablet for 3 replicas of tablet 27cacfdbe1c9487eb58856692ea19714
I20260926 07:05:37.040966 21275 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cacfdbe1c9487eb58856692ea19714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:52252
I20260926 07:05:37.041029 21141 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cacfdbe1c9487eb58856692ea19714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:51292
I20260926 07:05:37.041025 21409 tablet_service.cc:1514] Processing DeleteTablet for tablet 27cacfdbe1c9487eb58856692ea19714 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='kudu', principal='kudu/127.1.217.126@KRBTEST.COM'} at 127.0.0.1:35634
I20260926 07:05:37.041289 21513 tablet_replica.cc:330] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: stopping tablet replica
I20260926 07:05:37.041393 21513 raft_consensus.cc:2229] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:37.041469 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21078
I20260926 07:05:37.041513 21513 raft_consensus.cc:2258] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:37.043076 21511 tablet_replica.cc:330] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: stopping tablet replica
I20260926 07:05:37.043184 21511 raft_consensus.cc:2229] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:37.043286 21511 raft_consensus.cc:2258] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:37.043756 21511 ts_tablet_manager.cc:1905] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:37.044710 21513 ts_tablet_manager.cc:1905] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:37.045295 21511 ts_tablet_manager.cc:1918] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:37.045377 21511 log.cc:1196] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/27cacfdbe1c9487eb58856692ea19714
I20260926 07:05:37.045624 21511 ts_tablet_manager.cc:1939] T 27cacfdbe1c9487eb58856692ea19714 P b4ec577e9a5a452b8171d68b19a2e4d1: Deleting consensus metadata
I20260926 07:05:37.046005 21001 catalog_manager.cc:4841] TS b4ec577e9a5a452b8171d68b19a2e4d1 (127.1.217.66:43803): tablet 27cacfdbe1c9487eb58856692ea19714 (table test-table [id=3749fe7ad9404e758d5c8f28ed82503a]) successfully deleted
I20260926 07:05:37.046697 21513 ts_tablet_manager.cc:1918] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:37.046772 21513 log.cc:1196] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/27cacfdbe1c9487eb58856692ea19714
I20260926 07:05:37.047039 21513 ts_tablet_manager.cc:1939] T 27cacfdbe1c9487eb58856692ea19714 P fab2c2465da14cd78ca40aa2bfa1ab7d: Deleting consensus metadata
I20260926 07:05:37.047442 21002 catalog_manager.cc:4841] TS fab2c2465da14cd78ca40aa2bfa1ab7d (127.1.217.67:37229): tablet 27cacfdbe1c9487eb58856692ea19714 (table test-table [id=3749fe7ad9404e758d5c8f28ed82503a]) successfully deleted
W20260926 07:05:37.047768 21002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:44283 (error 108)
I20260926 07:05:37.048036 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21212
I20260926 07:05:37.048235 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.048087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:44283 (local address 127.0.0.1:51310)
0926 07:05:37.048156 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.048196 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:44283: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:05:37.048715 21002 catalog_manager.cc:4584] TS a386f1907d01464e9e647e6312bbe314 (127.1.217.65:44283): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27cacfdbe1c9487eb58856692ea19714: Network error: Client connection negotiation failed: client connection to 127.1.217.65:44283: connect: Connection refused (error 111)
I20260926 07:05:37.054811 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21346
I20260926 07:05:37.059223 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 20984
2026-09-26T07:05:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1901 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T07:05:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:37Z Disabled control of system clock
I20260926 07:05:37.086409 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37793
--webserver_interface=127.1.217.126
--webserver_port=0
--builtin_ntp_servers=127.1.217.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:37793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:37.166683 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:37.166891 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:37.166911 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:37.167985 21518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:37.168020 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:37.168032 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:37.168045 21518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:37.168057 21518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:37.169346 21518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:37793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:37793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:37.169544 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:37.169768 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:37.174392 21523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.174423 21518 server_base.cc:1034] running on GCE node
W20260926 07:05:37.174404 21524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:37.174404 21526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.174798 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:37.175123 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:37.176288 21518 hybrid_clock.cc:648] HybridClock initialized: now 1790406337176274 us; error 26 us; skew 500 ppm
I20260926 07:05:37.177534 21518 webserver.cc:460] Webserver started at http://127.1.217.126:38895/ using document root <none> and password file <none>
I20260926 07:05:37.177780 21518 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:37.177850 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:37.177980 21518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:37.178995 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "cc1f55ba0d2946508428ad45178f4785"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c78dcc0a6807a55d16718caafccfb899"
server_key_iv: "8801ac2512465777af51c6a66abed075"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.179361 21518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "cc1f55ba0d2946508428ad45178f4785"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c78dcc0a6807a55d16718caafccfb899"
server_key_iv: "8801ac2512465777af51c6a66abed075"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.180608 21518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:37.181373 21532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.181550 21518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.181659 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "cc1f55ba0d2946508428ad45178f4785"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c78dcc0a6807a55d16718caafccfb899"
server_key_iv: "8801ac2512465777af51c6a66abed075"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.181747 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:37.211038 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:37.211405 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:37.211568 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:37.215724 21518 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:37793
I20260926 07:05:37.215785 21584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:37793 every 8 connection(s)
I20260926 07:05:37.216143 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:37.217100 21585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:37.219413 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: Bootstrap starting.
I20260926 07:05:37.219998 21585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:37.220293 21585 log.cc:824] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:37.220695 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21518
I20260926 07:05:37.220841 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:37.221035 1893 external_mini_cluster.cc:1428] Setting key eda7e620422d8f773c5ba680d6e592b3
I20260926 07:05:37.221109 21585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: No bootstrap required, opened a new log
I20260926 07:05:37.223606 21585 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } }
I20260926 07:05:37.223795 21585 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:37.223836 21585 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f55ba0d2946508428ad45178f4785, State: Initialized, Role: FOLLOWER
I20260926 07:05:37.223956 21585 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } }
I20260926 07:05:37.224031 21585 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:37.224076 21585 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:37.224123 21585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.224845 21585 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } }
I20260926 07:05:37.224937 21585 leader_election.cc:302] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc1f55ba0d2946508428ad45178f4785; no voters:
I20260926 07:05:37.225116 21585 leader_election.cc:288] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:37.225196 21590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:37.225389 21590 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [term 1 LEADER]: Becoming Leader. State: Replica: cc1f55ba0d2946508428ad45178f4785, State: Running, Role: LEADER
I20260926 07:05:37.225389 21585 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:37.225560 21590 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } }
I20260926 07:05:37.225981 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc1f55ba0d2946508428ad45178f4785. Latest consensus state: current_term: 1 leader_uuid: "cc1f55ba0d2946508428ad45178f4785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } } }
I20260926 07:05:37.226066 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:37.226249 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc1f55ba0d2946508428ad45178f4785" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc1f55ba0d2946508428ad45178f4785" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 37793 } } }
I20260926 07:05:37.226567 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:37.226632 21594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:37.227138 21594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:37.227355 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.221567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51946 (local address 127.1.217.126:37793)
0926 07:05:37.221766 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.221773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.221797 (+ 24us) server_negotiation.cc:407] Connection header received
0926 07:05:37.222367 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.222373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.222431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.222545 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:37.222767 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.223635 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.224839 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.225527 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.226039 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.226470 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:37.226475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:37.226475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:37.226692 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:37.226736 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:37.226750 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:37.226813 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.227027 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.227122 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260926 07:05:37.228642 21594 catalog_manager.cc:1347] Generated new cluster ID: fc8c941601674684aebbe0166e2117c5
I20260926 07:05:37.228691 21594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:37.244454 21594 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:37.245187 21594 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:37.254251 21594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: Generated new TSK 0
I20260926 07:05:37.254464 21594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:37.257309 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.65:0
--local_ip_for_outbound_sockets=127.1.217.65
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--builtin_ntp_servers=127.1.217.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:37.350781 21609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:37.351032 21609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:37.351069 21609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:37.352231 21609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:37.352301 21609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:37.352345 21609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.65
I20260926 07:05:37.353839 21609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.1.217.65
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.65
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:37.354100 21609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:37.354413 21609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:37.360234 21615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:37.360366 21617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.360472 21609 server_base.cc:1034] running on GCE node
W20260926 07:05:37.360230 21614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.360714 21609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:37.361002 21609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:37.362166 21609 hybrid_clock.cc:648] HybridClock initialized: now 1790406337362155 us; error 35 us; skew 500 ppm
I20260926 07:05:37.363583 21609 webserver.cc:460] Webserver started at http://127.1.217.65:34661/ using document root <none> and password file <none>
I20260926 07:05:37.363867 21609 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:37.363942 21609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:37.364094 21609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:37.365229 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data/instance:
uuid: "0defa8d87776496291fc43c25d3b6e14"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c57c3986e454c050db5229b409e3736b"
server_key_iv: "12d799d6686352d40e9673c1fce495d6"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.365599 21609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance:
uuid: "0defa8d87776496291fc43c25d3b6e14"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c57c3986e454c050db5229b409e3736b"
server_key_iv: "12d799d6686352d40e9673c1fce495d6"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.366911 21609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:37.367671 21623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.367843 21609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:37.367949 21609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal
uuid: "0defa8d87776496291fc43c25d3b6e14"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "c57c3986e454c050db5229b409e3736b"
server_key_iv: "12d799d6686352d40e9673c1fce495d6"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.368017 21609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:37.377003 21609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:37.377269 21609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:37.377413 21609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:37.377616 21609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:37.377938 21609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:37.377995 21609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.378036 21609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:37.378062 21609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.383450 21609 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.65:35469
I20260926 07:05:37.383512 21736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.65:35469 every 8 connection(s)
I20260926 07:05:37.383878 21609 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/data/info.pb
I20260926 07:05:37.388985 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.384648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:59725 (local address 127.1.217.126:37793)
0926 07:05:37.384735 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.384738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.385023 (+ 285us) server_negotiation.cc:407] Connection header received
0926 07:05:37.385584 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.385585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.385599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.385659 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:37.386296 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.386792 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.387716 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.387896 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.387946 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.388774 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:37.388775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:37.388776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:37.388780 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:37.388790 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:37.388793 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:37.388830 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.388908 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.388928 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 07:05:37.389187 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.384635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37793 (local address 127.1.217.65:59725)
0926 07:05:37.384903 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.384937 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.385494 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.385713 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.385716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.385725 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:37.386213 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.386218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.386870 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.386871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.387640 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.387643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.387964 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.387964 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.388070 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.388639 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:37.388639 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:37.388667 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:37.388689 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:37.388833 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:37.388839 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:37.388845 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.388898 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.388978 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260926 07:05:37.389722 21737 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37793
I20260926 07:05:37.389835 21737 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:37.390021 21737 heartbeater.cc:511] Master 127.1.217.126:37793 requested a full tablet report, sending...
I20260926 07:05:37.390566 21549 ts_manager.cc:194] Registered new tserver with Master: 0defa8d87776496291fc43c25d3b6e14 (127.1.217.65:35469)
I20260926 07:05:37.391134 21549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.65:59725
I20260926 07:05:37.393929 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21609
I20260926 07:05:37.394024 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-0/wal/instance
I20260926 07:05:37.394251 1893 external_mini_cluster.cc:1428] Setting key ef5613acce7eea7af178039e23c95941
I20260926 07:05:37.395403 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.66:0
--local_ip_for_outbound_sockets=127.1.217.66
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--builtin_ntp_servers=127.1.217.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 07:05:37.476454 21740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:37.476643 21740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:37.476663 21740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:37.477784 21740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:37.477821 21740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:37.477845 21740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.66
I20260926 07:05:37.479107 21740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.1.217.66
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.66
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:37.479301 21740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:37.479571 21740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:37.484526 21746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:37.484603 21748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.484579 21740 server_base.cc:1034] running on GCE node
W20260926 07:05:37.484536 21745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.484985 21740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:37.485262 21740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:37.486430 21740 hybrid_clock.cc:648] HybridClock initialized: now 1790406337486420 us; error 41 us; skew 500 ppm
I20260926 07:05:37.487639 21740 webserver.cc:460] Webserver started at http://127.1.217.66:34117/ using document root <none> and password file <none>
I20260926 07:05:37.487869 21740 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:37.487936 21740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:37.488085 21740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:37.489214 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data/instance:
uuid: "e065daea2ae54161a52679b7a8d72c74"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "95e6db0752f2729966315b938fedc680"
server_key_iv: "4e5337a9c45e348de9f9a51cc148665e"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.489571 21740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance:
uuid: "e065daea2ae54161a52679b7a8d72c74"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "95e6db0752f2729966315b938fedc680"
server_key_iv: "4e5337a9c45e348de9f9a51cc148665e"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.490832 21740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:05:37.491503 21754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.491672 21740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:37.491782 21740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal
uuid: "e065daea2ae54161a52679b7a8d72c74"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "95e6db0752f2729966315b938fedc680"
server_key_iv: "4e5337a9c45e348de9f9a51cc148665e"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.491855 21740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:37.511989 21740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:37.512346 21740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:37.512511 21740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:37.512758 21740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:37.513136 21740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:37.513191 21740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.513260 21740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:37.513293 21740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.519035 21740 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.66:33395
I20260926 07:05:37.519074 21867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.66:33395 every 8 connection(s)
I20260926 07:05:37.519431 21740 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/data/info.pb
I20260926 07:05:37.520620 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21740
I20260926 07:05:37.520704 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal/instance
I20260926 07:05:37.520917 1893 external_mini_cluster.cc:1428] Setting key bfccf12d78d858b34c1b71b9a5c7ecaa
I20260926 07:05:37.521938 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.217.67:0
--local_ip_for_outbound_sockets=127.1.217.67
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--builtin_ntp_servers=127.1.217.84:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 07:05:37.525870 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.520236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.66:44203 (local address 127.1.217.126:37793)
0926 07:05:37.520305 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.520307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.520545 (+ 238us) server_negotiation.cc:407] Connection header received
0926 07:05:37.521247 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.521249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.521261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.521349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:37.522208 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.522707 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.524289 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.524481 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.524541 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.525491 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:37.525492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:37.525492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:37.525496 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:37.525511 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:37.525515 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:37.525600 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.525745 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.525791 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 07:05:37.526204 21870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.520220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37793 (local address 127.1.217.66:44203)
0926 07:05:37.520400 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.520443 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.521137 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.521425 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.521430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.521440 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:37.522106 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.522113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.522826 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.522827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.524188 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.524191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.524569 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.524570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.524737 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.525328 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:37.525329 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:37.525372 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:37.525405 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:37.525598 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:37.525602 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:37.525606 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.525733 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.525899 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260926 07:05:37.526736 21868 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37793
I20260926 07:05:37.526866 21868 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:37.527050 21868 heartbeater.cc:511] Master 127.1.217.126:37793 requested a full tablet report, sending...
I20260926 07:05:37.527405 21549 ts_manager.cc:194] Registered new tserver with Master: e065daea2ae54161a52679b7a8d72c74 (127.1.217.66:33395)
I20260926 07:05:37.527733 21549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.66:44203
W20260926 07:05:37.603014 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:37.603264 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:37.603302 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:37.604497 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:37.604565 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:37.604607 21871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.217.67
I20260926 07:05:37.605993 21871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.217.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.1.217.67
--webserver_port=0
--tserver_master_addrs=127.1.217.126:37793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.217.67
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:37.606274 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:37.606581 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:37.611562 21877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:37.611583 21876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:37.611583 21879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:37.611869 21871 server_base.cc:1034] running on GCE node
I20260926 07:05:37.612085 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:37.612339 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:37.613507 21871 hybrid_clock.cc:648] HybridClock initialized: now 1790406337613497 us; error 38 us; skew 500 ppm
I20260926 07:05:37.614794 21871 webserver.cc:460] Webserver started at http://127.1.217.67:40213/ using document root <none> and password file <none>
I20260926 07:05:37.615046 21871 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:37.615146 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:37.615375 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:37.616493 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data/instance:
uuid: "29672f9f5da14908800b550d57edcb8c"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "9637150194b183f6192cf28f4e851c48"
server_key_iv: "5581c838285b53cd7e880b99284619a0"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.616921 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance:
uuid: "29672f9f5da14908800b550d57edcb8c"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "9637150194b183f6192cf28f4e851c48"
server_key_iv: "5581c838285b53cd7e880b99284619a0"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.618211 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:37.618924 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.619102 21871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:37.619199 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal
uuid: "29672f9f5da14908800b550d57edcb8c"
format_stamp: "Formatted at 2026-09-26 07:05:37 on dist-test-slave-t1mc"
server_key: "9637150194b183f6192cf28f4e851c48"
server_key_iv: "5581c838285b53cd7e880b99284619a0"
server_key_version: "encryptionkey@0"
I20260926 07:05:37.619287 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:37.630035 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:37.630357 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:37.630507 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:37.630744 21871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 07:05:37.631062 21871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 07:05:37.631124 21871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.631166 21871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 07:05:37.631194 21871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:37.637143 21871 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.67:38811
I20260926 07:05:37.637218 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.67:38811 every 8 connection(s)
I20260926 07:05:37.637630 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/data/info.pb
I20260926 07:05:37.642797 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.638439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.67:37187 (local address 127.1.217.126:37793)
0926 07:05:37.638492 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.638494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.638748 (+ 254us) server_negotiation.cc:407] Connection header received
0926 07:05:37.639212 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.639213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.639223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.639280 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:37.640015 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.640498 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.641531 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.641727 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.641786 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.642564 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:37.642565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:37.642565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:37.642569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:37.642583 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:37.642597 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:37.642640 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.642716 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.642737 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:05:37.642982 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.638447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:37793 (local address 127.1.217.67:37187)
0926 07:05:37.638633 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.638666 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.639123 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.639333 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.639337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.639345 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:05:37.639833 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.639837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.640583 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.640583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.641445 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.641447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.641804 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.641805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.641909 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.642440 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 07:05:37.642440 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:05:37.642468 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 07:05:37.642490 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:05:37.642650 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:05:37.642653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:05:37.642656 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.642706 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.642783 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260926 07:05:37.643476 21999 heartbeater.cc:347] Connected to a master server at 127.1.217.126:37793
I20260926 07:05:37.643577 21999 heartbeater.cc:464] Registering TS with master...
I20260926 07:05:37.643712 21999 heartbeater.cc:511] Master 127.1.217.126:37793 requested a full tablet report, sending...
I20260926 07:05:37.644055 21549 ts_manager.cc:194] Registered new tserver with Master: 29672f9f5da14908800b550d57edcb8c (127.1.217.67:38811)
I20260926 07:05:37.644449 21549 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.217.67:37187
I20260926 07:05:37.647145 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 21871
I20260926 07:05:37.647240 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal/instance
I20260926 07:05:37.647385 1893 external_mini_cluster.cc:1428] Setting key bc1d3f2bbe9ba9dc3306d8a564af3662
I20260926 07:05:37.648265 1893 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 07:05:37.652671 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.649678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57630 (local address 127.1.217.126:37793)
0926 07:05:37.649773 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.649775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.649887 (+ 112us) server_negotiation.cc:407] Connection header received
0926 07:05:37.649965 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.649966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.649976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.650045 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:37.650213 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.650684 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.651438 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.651627 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.651699 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.652417 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:37.652418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:37.652419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:37.652423 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:37.652439 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:37.652448 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:37.652507 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.652563 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.652584 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 07:05:37.654233 21549 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 07:05:37.655776 21549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 07:05:37.661657 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.657963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:33395 (local address 127.0.0.1:36930)
0926 07:05:37.658121 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.658164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.658231 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.658458 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.658460 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.658462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.658527 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.658529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.659591 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.659592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.660560 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.660563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.661359 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.661360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.661437 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.661451 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.661530 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.661580 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260926 07:05:37.661665 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.658050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36930 (local address 127.1.217.66:33395)
0926 07:05:37.658223 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.658228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.658240 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:05:37.658337 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.658339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.658354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.658452 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.658614 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.659506 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.660644 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.661278 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.661353 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.661442 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.661544 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.661579 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260926 07:05:37.662407 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.657967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:35469 (local address 127.0.0.1:59914)
0926 07:05:37.658275 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.658287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.658338 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.658863 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.658864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.658866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.658910 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.658912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.660255 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.660256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.661180 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.661182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.662143 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.662144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.662216 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.662219 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.662294 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.662336 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":102,"threads_started":1}
I20260926 07:05:37.662424 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.658309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59914 (local address 127.1.217.65:35469)
0926 07:05:37.658471 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.658476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.658486 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:37.658528 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.658530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.658547 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.658994 (+ 447us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.659060 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.660165 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.661263 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.661965 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.662057 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.662148 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.662310 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.662348 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 07:05:37.662721 21802 tablet_service.cc:1467] Processing CreateTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 (DEFAULT_TABLE table=test-table [id=6e4c4559cc4c4303a5139cac5535958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:37.663112 21802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabd827e56344ab4b4a0baf4d20bbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:37.663456 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.658160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:38811 (local address 127.0.0.1:35932)
0926 07:05:37.658573 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.658583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.658629 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.658967 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.658968 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.658970 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.659008 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.659010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.661252 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.661253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.662338 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.662341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.663231 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.663232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.663293 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.663297 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.663342 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.663382 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":72,"threads_started":1}
I20260926 07:05:37.663482 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.658567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.1.217.67:38811)
0926 07:05:37.658751 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.658755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.658764 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:37.658810 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.658812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.658826 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.658903 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.659092 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.660166 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.662434 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.663138 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.663224 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.663320 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.663352 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.663391 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260926 07:05:37.663393 21671 tablet_service.cc:1467] Processing CreateTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 (DEFAULT_TABLE table=test-table [id=6e4c4559cc4c4303a5139cac5535958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:37.663712 21671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabd827e56344ab4b4a0baf4d20bbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:37.665395 21933 tablet_service.cc:1467] Processing CreateTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 (DEFAULT_TABLE table=test-table [id=6e4c4559cc4c4303a5139cac5535958c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 07:05:37.665733 21933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fabd827e56344ab4b4a0baf4d20bbb29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:37.666102 22013 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Bootstrap starting.
I20260926 07:05:37.666590 22014 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Bootstrap starting.
I20260926 07:05:37.666687 22013 tablet_bootstrap.cc:654] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:37.667013 22013 log.cc:824] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:37.667237 22014 tablet_bootstrap.cc:654] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:37.667511 22014 log.cc:824] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:37.667791 22013 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: No bootstrap required, opened a new log
I20260926 07:05:37.667836 22013 ts_tablet_manager.cc:1397] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:37.668270 22014 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: No bootstrap required, opened a new log
I20260926 07:05:37.668335 22014 ts_tablet_manager.cc:1397] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:37.668581 22017 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Bootstrap starting.
I20260926 07:05:37.669260 22017 tablet_bootstrap.cc:654] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:37.669610 22017 log.cc:824] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:37.670281 22017 tablet_bootstrap.cc:492] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: No bootstrap required, opened a new log
I20260926 07:05:37.670356 22017 ts_tablet_manager.cc:1397] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 07:05:37.670693 22013 raft_consensus.cc:348] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.670866 22013 raft_consensus.cc:374] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:37.670902 22013 raft_consensus.cc:729] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e065daea2ae54161a52679b7a8d72c74, State: Initialized, Role: FOLLOWER
I20260926 07:05:37.671059 22013 consensus_queue.cc:260] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.671051 22014 raft_consensus.cc:348] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.671324 22013 ts_tablet_manager.cc:1428] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 07:05:37.671314 22014 raft_consensus.cc:374] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:37.671367 22014 raft_consensus.cc:729] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0defa8d87776496291fc43c25d3b6e14, State: Initialized, Role: FOLLOWER
I20260926 07:05:37.671440 21868 heartbeater.cc:503] Master 127.1.217.126:37793 was elected leader, sending a full tablet report...
I20260926 07:05:37.671492 22014 consensus_queue.cc:260] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.671761 22014 ts_tablet_manager.cc:1428] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:05:37.671903 21737 heartbeater.cc:503] Master 127.1.217.126:37793 was elected leader, sending a full tablet report...
I20260926 07:05:37.673209 22017 raft_consensus.cc:348] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.673323 22017 raft_consensus.cc:374] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:37.673365 22017 raft_consensus.cc:729] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29672f9f5da14908800b550d57edcb8c, State: Initialized, Role: FOLLOWER
I20260926 07:05:37.673475 22017 consensus_queue.cc:260] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.673741 22017 ts_tablet_manager.cc:1428] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 07:05:37.673770 21999 heartbeater.cc:503] Master 127.1.217.126:37793 was elected leader, sending a full tablet report...
W20260926 07:05:37.770049 21869 tablet.cc:2369] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:37.883206 22020 raft_consensus.cc:482] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 07:05:37.883327 22020 raft_consensus.cc:504] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.883767 22020 leader_election.cc:288] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e065daea2ae54161a52679b7a8d72c74 (127.1.217.66:33395), 29672f9f5da14908800b550d57edcb8c (127.1.217.67:38811)
W20260926 07:05:37.884546 21738 tablet.cc:2369] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:37.887517 22012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.884216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:45443 (local address 127.1.217.67:38811)
0926 07:05:37.884263 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.884265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.884273 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:05:37.884300 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.884301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.884308 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.884392 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.884610 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.885316 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.886619 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.887173 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.887245 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.887253 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.887410 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.887449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 07:05:37.887517 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.884029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.217.65:33333 (local address 127.1.217.66:33395)
0926 07:05:37.884093 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.884096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.884106 (+ 10us) server_negotiation.cc:407] Connection header received
0926 07:05:37.884138 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.884139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.884148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.884220 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 07:05:37.884398 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.885220 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.886631 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.887166 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.887246 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.887253 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.887410 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.887449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 07:05:37.887534 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.883929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.66:33395 (local address 127.1.217.65:33333)
0926 07:05:37.883982 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.883994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.884038 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.884227 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.884228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.884230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.884321 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.884323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.885291 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.885292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.886525 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.886527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.887254 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.887255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.887329 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.887333 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.887393 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.887443 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":34}
I20260926 07:05:37.887540 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.884060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.67:38811 (local address 127.1.217.65:45443)
0926 07:05:37.884170 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.884178 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 07:05:37.884219 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:05:37.884399 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:05:37.884399 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:05:37.884401 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 07:05:37.884493 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.884495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.885384 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.885385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.886527 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:05:37.886528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.887256 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.887256 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:05:37.887341 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.887344 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 07:05:37.887394 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 07:05:37.887440 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":9,"thread_start_us":79,"threads_started":1}
I20260926 07:05:37.887957 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabd827e56344ab4b4a0baf4d20bbb29" candidate_uuid: "0defa8d87776496291fc43c25d3b6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e065daea2ae54161a52679b7a8d72c74" is_pre_election: true
I20260926 07:05:37.887953 21953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabd827e56344ab4b4a0baf4d20bbb29" candidate_uuid: "0defa8d87776496291fc43c25d3b6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29672f9f5da14908800b550d57edcb8c" is_pre_election: true
I20260926 07:05:37.888103 21822 raft_consensus.cc:2454] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0defa8d87776496291fc43c25d3b6e14 in term 0.
I20260926 07:05:37.888121 21953 raft_consensus.cc:2454] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0defa8d87776496291fc43c25d3b6e14 in term 0.
W20260926 07:05:37.888211 22000 tablet.cc:2369] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 07:05:37.888339 21627 leader_election.cc:302] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0defa8d87776496291fc43c25d3b6e14, e065daea2ae54161a52679b7a8d72c74; no voters:
I20260926 07:05:37.888451 22020 raft_consensus.cc:2781] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 07:05:37.888522 22020 raft_consensus.cc:482] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 07:05:37.888556 22020 raft_consensus.cc:3037] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.889341 22020 raft_consensus.cc:504] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.889474 22020 leader_election.cc:288] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [CANDIDATE]: Term 1 election: Requested vote from peers e065daea2ae54161a52679b7a8d72c74 (127.1.217.66:33395), 29672f9f5da14908800b550d57edcb8c (127.1.217.67:38811)
I20260926 07:05:37.889665 21953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabd827e56344ab4b4a0baf4d20bbb29" candidate_uuid: "0defa8d87776496291fc43c25d3b6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29672f9f5da14908800b550d57edcb8c"
I20260926 07:05:37.889664 21822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fabd827e56344ab4b4a0baf4d20bbb29" candidate_uuid: "0defa8d87776496291fc43c25d3b6e14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e065daea2ae54161a52679b7a8d72c74"
I20260926 07:05:37.889746 21822 raft_consensus.cc:3037] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.889745 21953 raft_consensus.cc:3037] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:37.890632 21953 raft_consensus.cc:2454] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0defa8d87776496291fc43c25d3b6e14 in term 1.
I20260926 07:05:37.890632 21822 raft_consensus.cc:2454] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0defa8d87776496291fc43c25d3b6e14 in term 1.
I20260926 07:05:37.890823 21627 leader_election.cc:302] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0defa8d87776496291fc43c25d3b6e14, 29672f9f5da14908800b550d57edcb8c; no voters:
I20260926 07:05:37.890909 22020 raft_consensus.cc:2781] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:37.891062 22020 raft_consensus.cc:686] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [term 1 LEADER]: Becoming Leader. State: Replica: 0defa8d87776496291fc43c25d3b6e14, State: Running, Role: LEADER
I20260926 07:05:37.891165 22020 consensus_queue.cc:237] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } }
I20260926 07:05:37.891992 21549 catalog_manager.cc:5495] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0defa8d87776496291fc43c25d3b6e14 (127.1.217.65). New cstate: current_term: 1 leader_uuid: "0defa8d87776496291fc43c25d3b6e14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0defa8d87776496291fc43c25d3b6e14" member_type: VOTER last_known_addr { host: "127.1.217.65" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 } health_report { overall_health: UNKNOWN } } }
I20260926 07:05:37.944149 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.941845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.1.217.65:35469)
0926 07:05:37.941916 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:37.941918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:37.941927 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:05:37.941985 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:37.941986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:37.942005 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:37.942066 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:05:37.942222 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.942682 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.943405 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:37.943580 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:37.943627 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:37.943768 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:05:37.943866 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:05:37.943926 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:37.943977 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 07:05:37.943997 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 07:05:37.946007 21822 raft_consensus.cc:1261] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 1 FOLLOWER]: Refusing update from remote peer 0defa8d87776496291fc43c25d3b6e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:37.946007 21953 raft_consensus.cc:1261] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 1 FOLLOWER]: Refusing update from remote peer 0defa8d87776496291fc43c25d3b6e14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:05:37.946365 22023 consensus_queue.cc:1035] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29672f9f5da14908800b550d57edcb8c" member_type: VOTER last_known_addr { host: "127.1.217.67" port: 38811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:37.946506 22020 consensus_queue.cc:1035] T fabd827e56344ab4b4a0baf4d20bbb29 P 0defa8d87776496291fc43c25d3b6e14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e065daea2ae54161a52679b7a8d72c74" member_type: VOTER last_known_addr { host: "127.1.217.66" port: 33395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:05:37.949921 22027 mvcc.cc:205] Tried to move back new op lower bound from 7333504360223989760 to 7333504360003051520. Current Snapshot: MvccSnapshot[applied={T|T < 7333504360223989760 or (T in {7333504360223989760})}]
I20260926 07:05:37.950018 22028 mvcc.cc:205] Tried to move back new op lower bound from 7333504360223989760 to 7333504360003051520. Current Snapshot: MvccSnapshot[applied={T|T < 7333504360223989760}]
I20260926 07:05:37.951113 21549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:37.951221 21549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 07:05:37.951947 21549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc1f55ba0d2946508428ad45178f4785: Sending DeleteTablet for 3 replicas of tablet fabd827e56344ab4b4a0baf4d20bbb29
I20260926 07:05:37.952260 21671 tablet_service.cc:1514] Processing DeleteTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='slave'} at 127.0.0.1:59914
I20260926 07:05:37.952288 21802 tablet_service.cc:1514] Processing DeleteTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='slave'} at 127.0.0.1:36930
I20260926 07:05:37.952514 22037 tablet_replica.cc:330] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: stopping tablet replica
I20260926 07:05:37.952625 22037 raft_consensus.cc:2229] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:37.952675 21933 tablet_service.cc:1514] Processing DeleteTablet for tablet fabd827e56344ab4b4a0baf4d20bbb29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 07:05:37 UTC) from {username='slave'} at 127.0.0.1:35932
I20260926 07:05:37.952733 22037 raft_consensus.cc:2258] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:37.952733 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21609
I20260926 07:05:37.953336 22038 tablet_replica.cc:330] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: stopping tablet replica
I20260926 07:05:37.953482 22038 raft_consensus.cc:2229] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 07:05:37.953675 22038 raft_consensus.cc:2258] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 07:05:37.954109 22038 ts_tablet_manager.cc:1905] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:37.955188 22037 ts_tablet_manager.cc:1905] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 07:05:37.955911 22038 ts_tablet_manager.cc:1918] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:37.956003 22038 log.cc:1196] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-2/wal/wals/fabd827e56344ab4b4a0baf4d20bbb29
I20260926 07:05:37.956312 22038 ts_tablet_manager.cc:1939] T fabd827e56344ab4b4a0baf4d20bbb29 P 29672f9f5da14908800b550d57edcb8c: Deleting consensus metadata
I20260926 07:05:37.956688 21536 catalog_manager.cc:4841] TS 29672f9f5da14908800b550d57edcb8c (127.1.217.67:38811): tablet fabd827e56344ab4b4a0baf4d20bbb29 (table test-table [id=6e4c4559cc4c4303a5139cac5535958c]) successfully deleted
I20260926 07:05:37.956991 22037 ts_tablet_manager.cc:1918] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 07:05:37.957053 22037 log.cc:1196] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Deleting WAL directory at /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790406101908495-1893-0/minicluster-data/ts-1/wal/wals/fabd827e56344ab4b4a0baf4d20bbb29
I20260926 07:05:37.957331 22037 ts_tablet_manager.cc:1939] T fabd827e56344ab4b4a0baf4d20bbb29 P e065daea2ae54161a52679b7a8d72c74: Deleting consensus metadata
I20260926 07:05:37.957720 21536 catalog_manager.cc:4841] TS e065daea2ae54161a52679b7a8d72c74 (127.1.217.66:33395): tablet fabd827e56344ab4b4a0baf4d20bbb29 (table test-table [id=6e4c4559cc4c4303a5139cac5535958c]) successfully deleted
W20260926 07:05:37.958551 21533 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.1.217.65:35469 (error 108)
I20260926 07:05:37.958760 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21740
I20260926 07:05:37.959332 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:37.958908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.65:35469 (local address 127.0.0.1:59934)
0926 07:05:37.959257 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 07:05:37.959299 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.65:35469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260926 07:05:37.959484 21533 catalog_manager.cc:4584] TS 0defa8d87776496291fc43c25d3b6e14 (127.1.217.65:35469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fabd827e56344ab4b4a0baf4d20bbb29: Network error: Client connection negotiation failed: client connection to 127.1.217.65:35469: connect: Connection refused (error 111)
I20260926 07:05:37.964221 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21871
I20260926 07:05:37.968411 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 21518
2026-09-26T07:05:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (908 ms)
[----------] 6 tests from EncryptionPolicyTest (8697 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:38 dist-test-slave-t1mc krb5kdc[22042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:38 dist-test-slave-t1mc krb5kdc[22042](info): set up 2 sockets
Sep 26 07:05:38 dist-test-slave-t1mc krb5kdc[22042](info): commencing operation
krb5kdc: starting...
W20260926 07:05:39.004899 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:39.123556 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:39.202742 22055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:39.202960 22055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:39.202983 22055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:39.204058 22055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:39.204095 22055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:39.204124 22055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:39.204137 22055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:39.204149 22055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:39.205637 22055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:39.205840 22055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:39.206003 22055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:05:39.211038 22055 server_base.cc:1034] running on GCE node
W20260926 07:05:39.210930 22063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:39.210933 22060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:39.210937 22061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:39.211376 22055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:39.211616 22055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:39.212836 22055 hybrid_clock.cc:648] HybridClock initialized: now 1790406339212820 us; error 27 us; skew 500 ppm
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:39.215315 22055 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:39.215788 22055 webserver.cc:460] Webserver started at http://127.0.0.1:43641/ using document root <none> and password file <none>
I20260926 07:05:39.216030 22055 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:39.216102 22055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:39.216254 22055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:39.217375 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d18fb8d0d46c41c8ad9b396f5cee617d"
format_stamp: "Formatted at 2026-09-26 07:05:39 on dist-test-slave-t1mc"
server_key: "f4e67635b1fcbdc71e3ade569e831895"
server_key_iv: "a1fcfff2d343711f03f3a0b83fdca9be"
server_key_version: "encryptionkey@0"
I20260926 07:05:39.217752 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d18fb8d0d46c41c8ad9b396f5cee617d"
format_stamp: "Formatted at 2026-09-26 07:05:39 on dist-test-slave-t1mc"
server_key: "f4e67635b1fcbdc71e3ade569e831895"
server_key_iv: "a1fcfff2d343711f03f3a0b83fdca9be"
server_key_version: "encryptionkey@0"
I20260926 07:05:39.219121 22055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:39.219842 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:39.220008 22055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:39.220111 22055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d18fb8d0d46c41c8ad9b396f5cee617d"
format_stamp: "Formatted at 2026-09-26 07:05:39 on dist-test-slave-t1mc"
server_key: "f4e67635b1fcbdc71e3ade569e831895"
server_key_iv: "a1fcfff2d343711f03f3a0b83fdca9be"
server_key_version: "encryptionkey@0"
I20260926 07:05:39.220185 22055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:39.236074 22055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:39.238595 22055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:39.238760 22055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:39.242990 22055 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46457
I20260926 07:05:39.243057 22122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46457 every 8 connection(s)
I20260926 07:05:39.243382 22055 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:39.244375 22123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:39.246745 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: Bootstrap starting.
I20260926 07:05:39.247334 22123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:39.247628 22123 log.cc:824] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:39.248234 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22055
I20260926 07:05:39.248344 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:39.248425 22123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: No bootstrap required, opened a new log
I20260926 07:05:39.248553 1893 external_mini_cluster.cc:1428] Setting key decc5c1f9bd697ed3410f47cb4a932bf
I20260926 07:05:39.250907 22123 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20260926 07:05:39.251044 22123 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:39.251066 22123 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18fb8d0d46c41c8ad9b396f5cee617d, State: Initialized, Role: FOLLOWER
I20260926 07:05:39.251199 22123 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20260926 07:05:39.251279 22123 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:39.251302 22123 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:39.251368 22123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:39.251950 22123 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20260926 07:05:39.252039 22123 leader_election.cc:302] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18fb8d0d46c41c8ad9b396f5cee617d; no voters:
I20260926 07:05:39.252223 22123 leader_election.cc:288] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:39.252390 22129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:39.252483 22123 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:39.252552 22129 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [term 1 LEADER]: Becoming Leader. State: Replica: d18fb8d0d46c41c8ad9b396f5cee617d, State: Running, Role: LEADER
I20260926 07:05:39.252683 22129 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } }
I20260926 07:05:39.253013 22129 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18fb8d0d46c41c8ad9b396f5cee617d. Latest consensus state: current_term: 1 leader_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } } }
I20260926 07:05:39.253023 22130 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18fb8d0d46c41c8ad9b396f5cee617d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46457 } } }
I20260926 07:05:39.253185 22130 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [sys.catalog]: This master's current role is: LEADER
W20260926 07:05:39.254375 22144 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:39.254434 22144 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:39.254585 22129 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:39.254717 22137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:39.254912 22137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:05:39.257153 22137 catalog_manager.cc:1347] Generated new cluster ID: cc42f7ddf0f94275883b128b1c371c0b
I20260926 07:05:39.257203 22137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:39.261261 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:39.249116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39795 (local address 127.0.0.1:46457)
0926 07:05:39.249290 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:39.249299 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:39.249345 (+ 46us) server_negotiation.cc:407] Connection header received
0926 07:05:39.249844 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:39.249850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:39.249897 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:39.250011 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:39.250790 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:39.252155 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:39.253356 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:39.253982 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:39.254104 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:39.257238 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:39.257242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:39.257243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:39.257256 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:39.259313 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:39.259771 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:39.259783 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:39.259784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:39.259895 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:39.260134 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:39.260135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:39.260136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:39.260472 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:39.260663 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:39.260753 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:05:39.260902 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260926 07:05:39.262280 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:39.271066 22127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:39.263823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56251 (local address 127.0.0.1:46457)
0926 07:05:39.263861 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:39.263862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:39.263990 (+ 128us) server_negotiation.cc:407] Connection header received
0926 07:05:39.264047 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:39.264048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:39.264059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:39.264108 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:39.264761 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:39.265525 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:39.266545 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:39.266867 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:39.266923 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:39.268302 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:39.268304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:39.268304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:39.268310 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:39.269850 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:39.270282 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:39.270283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:39.270283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:39.270335 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:39.270565 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:39.270566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:39.270566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:39.270683 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:39.270773 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:39.270920 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 07:05:39.270978 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:05:39.273275 22137 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:39.273752 22137 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:39.279668 22137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d18fb8d0d46c41c8ad9b396f5cee617d: Generated new TSK 0
I20260926 07:05:39.279855 22137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:39.285100 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22055
W20260926 07:05:39.289363 22040 connection.cc:535] client connection to 127.0.0.1:46457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46457): Connection reset by peer (error 104)
2026-09-26T07:05:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22154](info): set up 2 sockets
Sep 26 07:05:39 dist-test-slave-t1mc krb5kdc[22154](info): commencing operation
krb5kdc: starting...
W20260926 07:05:40.320928 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:40 dist-test-slave-t1mc krb5kdc[22154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:40.440279 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:40.520207 22167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:40.520416 22167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:40.520439 22167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:40.521564 22167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:40.521601 22167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:40.521615 22167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:40.521628 22167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:40.521641 22167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:40.523216 22167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:40.523416 22167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:40.523586 22167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:40.528808 22173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:40.528838 22172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:40.528923 22167 server_base.cc:1034] running on GCE node
W20260926 07:05:40.528828 22175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:40.529210 22167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:40.529588 22167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:40.530790 22167 hybrid_clock.cc:648] HybridClock initialized: now 1790406340530784 us; error 38 us; skew 500 ppm
Sep 26 07:05:40 dist-test-slave-t1mc krb5kdc[22154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:40.537624 22167 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:40.538141 22167 webserver.cc:460] Webserver started at http://127.0.0.1:44307/ using document root <none> and password file <none>
I20260926 07:05:40.538440 22167 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:40.538512 22167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:40.538671 22167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:40.539783 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "c33de9fd44244ce696d98671dd1445a3"
format_stamp: "Formatted at 2026-09-26 07:05:40 on dist-test-slave-t1mc"
server_key: "2b1c07766fd7cc0f393ad751c2206867"
server_key_iv: "413913912a8cc70488262753d706b0ba"
server_key_version: "encryptionkey@0"
I20260926 07:05:40.540158 22167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "c33de9fd44244ce696d98671dd1445a3"
format_stamp: "Formatted at 2026-09-26 07:05:40 on dist-test-slave-t1mc"
server_key: "2b1c07766fd7cc0f393ad751c2206867"
server_key_iv: "413913912a8cc70488262753d706b0ba"
server_key_version: "encryptionkey@0"
I20260926 07:05:40.541664 22167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 07:05:40.542496 22182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:40.542721 22167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:40.542840 22167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "c33de9fd44244ce696d98671dd1445a3"
format_stamp: "Formatted at 2026-09-26 07:05:40 on dist-test-slave-t1mc"
server_key: "2b1c07766fd7cc0f393ad751c2206867"
server_key_iv: "413913912a8cc70488262753d706b0ba"
server_key_version: "encryptionkey@0"
I20260926 07:05:40.542919 22167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:40.561352 22167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:40.563373 22167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:40.563565 22167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:40.567627 22167 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34225
I20260926 07:05:40.567680 22234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34225 every 8 connection(s)
I20260926 07:05:40.568046 22167 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:40.569084 22235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:40.571367 22235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3: Bootstrap starting.
I20260926 07:05:40.571957 22235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:40.572249 22235 log.cc:824] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:40.573036 22235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3: No bootstrap required, opened a new log
I20260926 07:05:40.574600 22235 raft_consensus.cc:348] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } }
I20260926 07:05:40.574738 22235 raft_consensus.cc:374] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:40.574759 22235 raft_consensus.cc:729] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33de9fd44244ce696d98671dd1445a3, State: Initialized, Role: FOLLOWER
I20260926 07:05:40.574965 22235 consensus_queue.cc:260] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } }
I20260926 07:05:40.575026 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22167
I20260926 07:05:40.575069 22235 raft_consensus.cc:388] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:40.575109 22235 raft_consensus.cc:482] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:40.575148 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:40.575160 22235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:40.575335 1893 external_mini_cluster.cc:1428] Setting key 01362d5c45fde6251310fd7be80a424d
I20260926 07:05:40.575802 22235 raft_consensus.cc:504] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } }
I20260926 07:05:40.575939 22235 leader_election.cc:302] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c33de9fd44244ce696d98671dd1445a3; no voters:
I20260926 07:05:40.576122 22235 leader_election.cc:288] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:40.576149 22240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:40.576252 22240 raft_consensus.cc:686] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [term 1 LEADER]: Becoming Leader. State: Replica: c33de9fd44244ce696d98671dd1445a3, State: Running, Role: LEADER
I20260926 07:05:40.576336 22240 consensus_queue.cc:237] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } }
I20260926 07:05:40.576401 22235 sys_catalog.cc:564] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:40.576642 22240 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c33de9fd44244ce696d98671dd1445a3. Latest consensus state: current_term: 1 leader_uuid: "c33de9fd44244ce696d98671dd1445a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } } }
I20260926 07:05:40.576699 22240 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:40.576959 22245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:40.577790 22242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c33de9fd44244ce696d98671dd1445a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c33de9fd44244ce696d98671dd1445a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34225 } } }
I20260926 07:05:40.577884 22242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:40.577975 22245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:40.579950 22245 catalog_manager.cc:1347] Generated new cluster ID: eea7357bd0bd499982eac27aef44cf77
I20260926 07:05:40.580006 22245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:40.582304 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:40.575829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51533 (local address 127.0.0.1:34225)
0926 07:05:40.576057 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:40.576067 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:40.576105 (+ 38us) server_negotiation.cc:407] Connection header received
0926 07:05:40.576238 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:40.576246 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:40.576305 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:40.576483 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:40.576703 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:40.578981 (+ 2278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:40.579984 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:40.580590 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:40.580721 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:40.581521 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:40.581525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:40.581525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:40.581754 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:40.581825 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:40.581842 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:40.581942 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:40.582024 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 07:05:40.582058 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260926 07:05:40.583308 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:40.588635 22239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:40.584894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57529 (local address 127.0.0.1:34225)
0926 07:05:40.584931 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:40.584933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:40.585044 (+ 111us) server_negotiation.cc:407] Connection header received
0926 07:05:40.585148 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:40.585149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:40.585161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:40.585210 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:40.585369 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:40.586095 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:40.587205 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:40.587512 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:40.587569 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:40.588358 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:40.588360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:40.588360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:40.588365 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:40.588380 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:40.588385 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:40.588439 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:40.588544 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 07:05:40.588555 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:05:40.618796 22245 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:40.619360 22245 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:40.624442 22245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c33de9fd44244ce696d98671dd1445a3: Generated new TSK 0
I20260926 07:05:40.624670 22245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:40.627575 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22167
W20260926 07:05:40.632134 22152 connection.cc:535] client connection to 127.0.0.1:34225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34225): Connection reset by peer (error 104)
2026-09-26T07:05:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:40 dist-test-slave-t1mc krb5kdc[22266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:40 dist-test-slave-t1mc krb5kdc[22266](info): set up 2 sockets
Sep 26 07:05:40 dist-test-slave-t1mc krb5kdc[22266](info): commencing operation
krb5kdc: starting...
W20260926 07:05:41.660676 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:41 dist-test-slave-t1mc krb5kdc[22266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:41.775967 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:41.855239 22279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:41.855463 22279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:41.855499 22279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:41.856633 22279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:41.856702 22279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:41.856726 22279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:41.856747 22279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:41.856796 22279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:41.858295 22279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42969
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:41.858561 22279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:41.858789 22279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:41.864058 22284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:41.864125 22285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:41.864270 22279 server_base.cc:1034] running on GCE node
W20260926 07:05:41.864065 22287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:41.864599 22279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:41.864939 22279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:41.866089 22279 hybrid_clock.cc:648] HybridClock initialized: now 1790406341866067 us; error 37 us; skew 500 ppm
Sep 26 07:05:41 dist-test-slave-t1mc krb5kdc[22266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:41.868475 22279 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:41.868960 22279 webserver.cc:460] Webserver started at http://127.0.0.1:45759/ using document root <none> and password file <none>
I20260926 07:05:41.869199 22279 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:41.869267 22279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:41.869421 22279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:41.870477 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "05af42f17106453a96ae93fcf8949fac"
format_stamp: "Formatted at 2026-09-26 07:05:41 on dist-test-slave-t1mc"
server_key: "454e67ee414928b42f545376d2ee7943"
server_key_iv: "7d2df6a426795a7c4416161f7b941c98"
server_key_version: "encryptionkey@0"
I20260926 07:05:41.870836 22279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "05af42f17106453a96ae93fcf8949fac"
format_stamp: "Formatted at 2026-09-26 07:05:41 on dist-test-slave-t1mc"
server_key: "454e67ee414928b42f545376d2ee7943"
server_key_iv: "7d2df6a426795a7c4416161f7b941c98"
server_key_version: "encryptionkey@0"
I20260926 07:05:41.872082 22279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:41.872850 22294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:41.873025 22279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:41.873121 22279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "05af42f17106453a96ae93fcf8949fac"
format_stamp: "Formatted at 2026-09-26 07:05:41 on dist-test-slave-t1mc"
server_key: "454e67ee414928b42f545376d2ee7943"
server_key_iv: "7d2df6a426795a7c4416161f7b941c98"
server_key_version: "encryptionkey@0"
I20260926 07:05:41.873207 22279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:41.876298 22279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:41.878551 22279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:41.878700 22279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:41.882997 22279 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42969
I20260926 07:05:41.883026 22346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42969 every 8 connection(s)
I20260926 07:05:41.883347 22279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:41.884248 22347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:41.886474 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac: Bootstrap starting.
I20260926 07:05:41.887037 22347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:41.887312 22347 log.cc:824] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:41.887996 22347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac: No bootstrap required, opened a new log
I20260926 07:05:41.889612 22347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260926 07:05:41.889743 22347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:41.889766 22347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05af42f17106453a96ae93fcf8949fac, State: Initialized, Role: FOLLOWER
I20260926 07:05:41.889930 22347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260926 07:05:41.889992 22347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:41.890035 22347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:41.890046 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22279
I20260926 07:05:41.890082 22347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:41.890179 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:41.890398 1893 external_mini_cluster.cc:1428] Setting key 6f644dc46b63029e057e795cf8c45369
I20260926 07:05:41.890748 22347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260926 07:05:41.890894 22347 leader_election.cc:302] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05af42f17106453a96ae93fcf8949fac; no voters:
I20260926 07:05:41.891065 22347 leader_election.cc:288] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:41.891093 22352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:41.891302 22352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [term 1 LEADER]: Becoming Leader. State: Replica: 05af42f17106453a96ae93fcf8949fac, State: Running, Role: LEADER
I20260926 07:05:41.891464 22352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } }
I20260926 07:05:41.891337 22347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:41.892030 22354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05af42f17106453a96ae93fcf8949fac. Latest consensus state: current_term: 1 leader_uuid: "05af42f17106453a96ae93fcf8949fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260926 07:05:41.892258 22354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:41.892498 22358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:41.892822 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:41.890894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46891 (local address 127.0.0.1:42969)
0926 07:05:41.891077 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:41.891087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:41.891120 (+ 33us) server_negotiation.cc:407] Connection header received
0926 07:05:41.891241 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:41.891248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:41.891277 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:41.891454 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:41.892158 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:41.892161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:41.892161 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:41.892166 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:41.892220 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:41.892234 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:41.892301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:41.892549 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 07:05:41.892597 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260926 07:05:41.893045 22358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:41.893576 22353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05af42f17106453a96ae93fcf8949fac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05af42f17106453a96ae93fcf8949fac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42969 } } }
I20260926 07:05:41.893779 22353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:41.893975 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:41.894560 22358 catalog_manager.cc:1347] Generated new cluster ID: c8bb4ed31808478e9d513520ae3ea3c4
I20260926 07:05:41.894600 22358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:41.896488 22351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:41.895429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35129 (local address 127.0.0.1:42969)
0926 07:05:41.895461 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:41.895463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:41.895589 (+ 126us) server_negotiation.cc:407] Connection header received
0926 07:05:41.895630 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:41.895630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:41.895635 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:41.895686 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:41.896281 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:41.896282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:41.896282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:41.896286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:41.896300 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:41.896304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:41.896366 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:41.896419 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:05:41.896431 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:05:41.905107 22358 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:41.905269 22358 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:41.914635 22358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05af42f17106453a96ae93fcf8949fac: Generated new TSK 0
I20260926 07:05:41.914819 22358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:41.917832 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22279
2026-09-26T07:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1289 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:41 dist-test-slave-t1mc krb5kdc[22378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:41 dist-test-slave-t1mc krb5kdc[22378](info): set up 2 sockets
Sep 26 07:05:41 dist-test-slave-t1mc krb5kdc[22378](info): commencing operation
krb5kdc: starting...
W20260926 07:05:42.960906 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:43.083313 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:43.164376 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:43.164633 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:43.164675 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:43.165910 22391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:43.165987 22391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:43.166011 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:43.166033 22391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:43.166062 22391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:43.167532 22391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:43.167814 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:43.168040 22391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:43.172873 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:43.172905 22397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:43.172916 22399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:43.173085 22391 server_base.cc:1034] running on GCE node
I20260926 07:05:43.173396 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:43.173709 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:43.174858 22391 hybrid_clock.cc:648] HybridClock initialized: now 1790406343174840 us; error 34 us; skew 500 ppm
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:43.177433 22391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:43.177826 22391 webserver.cc:460] Webserver started at http://127.0.0.1:46721/ using document root <none> and password file <none>
I20260926 07:05:43.178006 22391 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:43.178045 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:43.178130 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:43.179155 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "f87c323b816245acba13635efc1c95d3"
format_stamp: "Formatted at 2026-09-26 07:05:43 on dist-test-slave-t1mc"
server_key: "f5dbc111e9c089ec18465c4d9cf7a3a5"
server_key_iv: "3688f0222217df86ceaa7fd89906e0b6"
server_key_version: "encryptionkey@0"
I20260926 07:05:43.179453 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "f87c323b816245acba13635efc1c95d3"
format_stamp: "Formatted at 2026-09-26 07:05:43 on dist-test-slave-t1mc"
server_key: "f5dbc111e9c089ec18465c4d9cf7a3a5"
server_key_iv: "3688f0222217df86ceaa7fd89906e0b6"
server_key_version: "encryptionkey@0"
I20260926 07:05:43.180691 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:43.181493 22406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:43.181648 22391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:43.181730 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "f87c323b816245acba13635efc1c95d3"
format_stamp: "Formatted at 2026-09-26 07:05:43 on dist-test-slave-t1mc"
server_key: "f5dbc111e9c089ec18465c4d9cf7a3a5"
server_key_iv: "3688f0222217df86ceaa7fd89906e0b6"
server_key_version: "encryptionkey@0"
I20260926 07:05:43.181785 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:43.193620 22391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:43.195629 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:43.195739 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:43.199579 22391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41713
I20260926 07:05:43.199633 22458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41713 every 8 connection(s)
I20260926 07:05:43.199976 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:43.201077 22459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:43.203498 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3: Bootstrap starting.
I20260926 07:05:43.204099 22459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:43.204420 22459 log.cc:824] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:43.205204 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3: No bootstrap required, opened a new log
I20260926 07:05:43.206916 22459 raft_consensus.cc:348] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260926 07:05:43.207086 22459 raft_consensus.cc:374] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:43.207139 22459 raft_consensus.cc:729] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f87c323b816245acba13635efc1c95d3, State: Initialized, Role: FOLLOWER
I20260926 07:05:43.207300 22459 consensus_queue.cc:260] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260926 07:05:43.207389 22459 raft_consensus.cc:388] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:43.207448 22459 raft_consensus.cc:482] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:43.207540 22459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:43.207942 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22391
I20260926 07:05:43.208073 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:43.208299 1893 external_mini_cluster.cc:1428] Setting key dff1eb3bc3eaa3c6326c7667b6dd898f
I20260926 07:05:43.208338 22459 raft_consensus.cc:504] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260926 07:05:43.208452 22459 leader_election.cc:302] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f87c323b816245acba13635efc1c95d3; no voters:
I20260926 07:05:43.208652 22459 leader_election.cc:288] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:43.208676 22462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:43.208961 22462 raft_consensus.cc:686] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [term 1 LEADER]: Becoming Leader. State: Replica: f87c323b816245acba13635efc1c95d3, State: Running, Role: LEADER
I20260926 07:05:43.209029 22459 sys_catalog.cc:564] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:43.209071 22462 consensus_queue.cc:237] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } }
I20260926 07:05:43.209435 22462 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f87c323b816245acba13635efc1c95d3. Latest consensus state: current_term: 1 leader_uuid: "f87c323b816245acba13635efc1c95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260926 07:05:43.209455 22466 sys_catalog.cc:455] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f87c323b816245acba13635efc1c95d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f87c323b816245acba13635efc1c95d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41713 } } }
I20260926 07:05:43.209550 22462 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:43.209617 22466 sys_catalog.cc:458] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:43.210126 22468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:43.210639 22468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:43.212529 22468 catalog_manager.cc:1347] Generated new cluster ID: 4076eab1e06e4668be592366b81ffe48
I20260926 07:05:43.212581 22468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:05:43.221108 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:43.208829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44121 (local address 127.0.0.1:41713)
0926 07:05:43.211016 (+ 2187us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:43.211026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:43.211065 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:05:43.211178 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:43.211184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:43.211270 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:43.211429 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:43.212186 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:43.213436 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:43.214341 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:43.214807 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:43.214944 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:43.217529 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:43.217533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:43.217533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:43.217554 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:43.219373 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:43.219803 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:43.219806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:43.219806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:43.219923 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:43.220154 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:43.220156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:43.220156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:43.220419 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:43.220596 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:43.220700 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 07:05:43.220899 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2088,"thread_start_us":77,"threads_started":1}
I20260926 07:05:43.222021 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:43.232127 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:43.225079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50521 (local address 127.0.0.1:41713)
0926 07:05:43.225119 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:43.225121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:43.225456 (+ 335us) server_negotiation.cc:407] Connection header received
0926 07:05:43.225550 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:43.225551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:43.225563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:43.225624 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:43.226188 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:43.226745 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:43.227535 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:43.227778 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:43.227870 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:43.229346 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:43.229347 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:43.229348 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:43.229354 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:43.230902 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:43.231297 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:43.231299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:43.231299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:43.231359 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:43.231573 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:43.231574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:43.231574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:43.231690 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:43.231809 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:43.231946 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 07:05:43.232023 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:05:43.243696 22468 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:43.244244 22468 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:43.251021 22468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f87c323b816245acba13635efc1c95d3: Generated new TSK 0
I20260926 07:05:43.251227 22468 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:43.258791 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22391
2026-09-26T07:05:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22490](info): set up 2 sockets
Sep 26 07:05:43 dist-test-slave-t1mc krb5kdc[22490](info): commencing operation
krb5kdc: starting...
W20260926 07:05:44.288983 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:44 dist-test-slave-t1mc krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:44.406796 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:44.487373 22503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:44.487582 22503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:44.487605 22503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:44.488687 22503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:44.488722 22503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:44.488737 22503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:44.488750 22503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:44.488762 22503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:44.490167 22503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:44.490358 22503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:44.490520 22503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:44.495456 22509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:44.495477 22511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:44.495455 22508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:44.495882 22503 server_base.cc:1034] running on GCE node
I20260926 07:05:44.496110 22503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:44.496402 22503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:44.497565 22503 hybrid_clock.cc:648] HybridClock initialized: now 1790406344497567 us; error 44 us; skew 500 ppm
Sep 26 07:05:44 dist-test-slave-t1mc krb5kdc[22490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:44.500022 22503 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:44.500396 22503 webserver.cc:460] Webserver started at http://127.0.0.1:35401/ using document root <none> and password file <none>
I20260926 07:05:44.500572 22503 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:44.500651 22503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:44.500758 22503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:44.501770 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "a1ee72b6f2af4838a977b321f7f7fc92"
format_stamp: "Formatted at 2026-09-26 07:05:44 on dist-test-slave-t1mc"
server_key: "7533f512221c1110b8fd71de5def5858"
server_key_iv: "a9ea2d3114144f2ca4bf16546c17b0dc"
server_key_version: "encryptionkey@0"
I20260926 07:05:44.502070 22503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "a1ee72b6f2af4838a977b321f7f7fc92"
format_stamp: "Formatted at 2026-09-26 07:05:44 on dist-test-slave-t1mc"
server_key: "7533f512221c1110b8fd71de5def5858"
server_key_iv: "a9ea2d3114144f2ca4bf16546c17b0dc"
server_key_version: "encryptionkey@0"
I20260926 07:05:44.503290 22503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:05:44.503961 22518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:44.504127 22503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:44.504221 22503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "a1ee72b6f2af4838a977b321f7f7fc92"
format_stamp: "Formatted at 2026-09-26 07:05:44 on dist-test-slave-t1mc"
server_key: "7533f512221c1110b8fd71de5def5858"
server_key_iv: "a9ea2d3114144f2ca4bf16546c17b0dc"
server_key_version: "encryptionkey@0"
I20260926 07:05:44.504313 22503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:44.546533 22503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:44.548950 22503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:44.549109 22503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:44.553071 22503 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41143
I20260926 07:05:44.553117 22570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41143 every 8 connection(s)
I20260926 07:05:44.553483 22503 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:44.554410 22571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:44.556660 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92: Bootstrap starting.
I20260926 07:05:44.557272 22571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:44.557562 22571 log.cc:824] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:44.558239 22571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92: No bootstrap required, opened a new log
I20260926 07:05:44.559811 22571 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260926 07:05:44.559944 22571 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:44.559968 22571 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1ee72b6f2af4838a977b321f7f7fc92, State: Initialized, Role: FOLLOWER
I20260926 07:05:44.560177 22571 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260926 07:05:44.560242 22571 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:44.560262 22571 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:44.560314 22571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:44.560842 22571 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260926 07:05:44.560954 22571 leader_election.cc:302] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1ee72b6f2af4838a977b321f7f7fc92; no voters:
I20260926 07:05:44.561142 22571 leader_election.cc:288] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:44.561174 22574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:44.561417 22574 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [term 1 LEADER]: Becoming Leader. State: Replica: a1ee72b6f2af4838a977b321f7f7fc92, State: Running, Role: LEADER
I20260926 07:05:44.561499 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22503
I20260926 07:05:44.561542 22571 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:44.561559 22574 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } }
I20260926 07:05:44.561615 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:44.561794 1893 external_mini_cluster.cc:1428] Setting key 5f19df3808363b3a92d75bf477c57272
I20260926 07:05:44.562016 22575 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260926 07:05:44.562099 22575 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:44.562305 22576 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1ee72b6f2af4838a977b321f7f7fc92. Latest consensus state: current_term: 1 leader_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1ee72b6f2af4838a977b321f7f7fc92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41143 } } }
I20260926 07:05:44.562384 22576 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:44.562748 22583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:44.563470 22583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:44.565084 22583 catalog_manager.cc:1347] Generated new cluster ID: 869ee49774104afba5c178616bef5450
I20260926 07:05:44.565136 22583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:44.567528 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:44.562563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36101 (local address 127.0.0.1:41143)
0926 07:05:44.562733 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:44.562743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:44.562779 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:05:44.562904 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:44.562912 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:44.562984 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:44.563187 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:44.563400 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:44.564642 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:44.565624 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:44.566079 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:44.566210 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:44.566888 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:44.566891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:44.566892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:44.567072 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:44.567102 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:44.567113 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:44.567185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:44.567258 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:05:44.567325 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260926 07:05:44.568372 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:44.572305 22583 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:44.573015 22583 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:44.575204 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:44.572182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60477 (local address 127.0.0.1:41143)
0926 07:05:44.572214 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:44.572216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:44.572431 (+ 215us) server_negotiation.cc:407] Connection header received
0926 07:05:44.572512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:44.572513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:44.572524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:44.572578 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:44.572700 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:44.573302 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:44.574002 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:44.574200 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:44.574270 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:44.574937 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:44.574939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:44.574939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:44.574943 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:44.574956 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:44.574960 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:44.575037 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:44.575094 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:05:44.575131 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:05:44.583695 22583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1ee72b6f2af4838a977b321f7f7fc92: Generated new TSK 0
I20260926 07:05:44.583873 22583 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:44.588627 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22503
2026-09-26T07:05:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:44 dist-test-slave-t1mc krb5kdc[22602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:44 dist-test-slave-t1mc krb5kdc[22602](info): set up 2 sockets
Sep 26 07:05:44 dist-test-slave-t1mc krb5kdc[22602](info): commencing operation
krb5kdc: starting...
W20260926 07:05:45.612960 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:45 dist-test-slave-t1mc krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:45.731551 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:45.812064 22615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:45.812283 22615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:45.812352 22615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:45.813505 22615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:45.813546 22615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:45.813560 22615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:45.813573 22615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:45.813585 22615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:45.814970 22615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:45.815158 22615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:45.815330 22615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:45.820223 22623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:45.820235 22621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:45.820276 22615 server_base.cc:1034] running on GCE node
W20260926 07:05:45.820231 22620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:45.820736 22615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:45.821053 22615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:45.822245 22615 hybrid_clock.cc:648] HybridClock initialized: now 1790406345822231 us; error 29 us; skew 500 ppm
Sep 26 07:05:45 dist-test-slave-t1mc krb5kdc[22602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:45.828956 22615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:45.829350 22615 webserver.cc:460] Webserver started at http://127.0.0.1:34993/ using document root <none> and password file <none>
I20260926 07:05:45.829593 22615 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:45.829664 22615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:45.829815 22615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:45.830914 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "c380108fba594631a09ceb5572bfbaca"
format_stamp: "Formatted at 2026-09-26 07:05:45 on dist-test-slave-t1mc"
server_key: "3a338a5a71eba8cfaa0f112baaac9406"
server_key_iv: "6bf42bbeba91b1f9d09e2b2274ae7927"
server_key_version: "encryptionkey@0"
I20260926 07:05:45.831286 22615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "c380108fba594631a09ceb5572bfbaca"
format_stamp: "Formatted at 2026-09-26 07:05:45 on dist-test-slave-t1mc"
server_key: "3a338a5a71eba8cfaa0f112baaac9406"
server_key_iv: "6bf42bbeba91b1f9d09e2b2274ae7927"
server_key_version: "encryptionkey@0"
I20260926 07:05:45.832664 22615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:45.833498 22630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:45.833706 22615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:45.833793 22615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "c380108fba594631a09ceb5572bfbaca"
format_stamp: "Formatted at 2026-09-26 07:05:45 on dist-test-slave-t1mc"
server_key: "3a338a5a71eba8cfaa0f112baaac9406"
server_key_iv: "6bf42bbeba91b1f9d09e2b2274ae7927"
server_key_version: "encryptionkey@0"
I20260926 07:05:45.833860 22615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:45.837153 22615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:45.838995 22615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:45.839097 22615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:45.842765 22615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40097
I20260926 07:05:45.842796 22682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40097 every 8 connection(s)
I20260926 07:05:45.843127 22615 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:45.844084 22683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:45.845992 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22615
I20260926 07:05:45.846103 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:45.846284 1893 external_mini_cluster.cc:1428] Setting key 1019a0705bc182e580253b018086be2c
I20260926 07:05:45.846683 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca: Bootstrap starting.
I20260926 07:05:45.847410 22683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:45.847721 22683 log.cc:824] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:45.848657 22683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca: No bootstrap required, opened a new log
I20260926 07:05:45.849108 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:45.846792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52063 (local address 127.0.0.1:40097)
0926 07:05:45.847108 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:45.847118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:45.847155 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:05:45.847760 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:45.847768 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:45.847803 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:45.847964 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:45.848597 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:45.848600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:45.848601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:45.848605 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:45.848647 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:45.848661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:45.848753 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:45.848883 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 07:05:45.848922 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":73,"threads_started":1}
I20260926 07:05:45.850190 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:45.851155 22683 raft_consensus.cc:348] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260926 07:05:45.851414 22683 raft_consensus.cc:374] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:45.851481 22683 raft_consensus.cc:729] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c380108fba594631a09ceb5572bfbaca, State: Initialized, Role: FOLLOWER
I20260926 07:05:45.851606 22683 consensus_queue.cc:260] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260926 07:05:45.851720 22683 raft_consensus.cc:388] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:45.851758 22683 raft_consensus.cc:482] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:45.851847 22683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:45.852345 22683 raft_consensus.cc:504] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260926 07:05:45.852428 22683 leader_election.cc:302] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c380108fba594631a09ceb5572bfbaca; no voters:
I20260926 07:05:45.852548 22683 leader_election.cc:288] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:45.852602 22693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:45.852794 22693 raft_consensus.cc:686] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [term 1 LEADER]: Becoming Leader. State: Replica: c380108fba594631a09ceb5572bfbaca, State: Running, Role: LEADER
I20260926 07:05:45.852818 22683 sys_catalog.cc:564] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:45.852900 22693 consensus_queue.cc:237] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260926 07:05:45.853291 22694 sys_catalog.cc:455] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c380108fba594631a09ceb5572bfbaca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260926 07:05:45.853302 22695 sys_catalog.cc:455] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [sys.catalog]: SysCatalogTable state changed. Reason: New leader c380108fba594631a09ceb5572bfbaca. Latest consensus state: current_term: 1 leader_uuid: "c380108fba594631a09ceb5572bfbaca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c380108fba594631a09ceb5572bfbaca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260926 07:05:45.853459 22694 sys_catalog.cc:458] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:45.853468 22695 sys_catalog.cc:458] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:45.854082 22702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:45.854604 22702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:45.855383 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:45.851759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41973 (local address 127.0.0.1:40097)
0926 07:05:45.851792 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:45.851794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:45.854376 (+ 2582us) server_negotiation.cc:407] Connection header received
0926 07:05:45.854460 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:45.854461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:45.854467 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:45.854523 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:45.855153 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:45.855155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:45.855155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:45.855160 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:45.855176 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:45.855181 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:45.855240 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:45.855304 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:05:45.855317 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:05:45.856488 22702 catalog_manager.cc:1347] Generated new cluster ID: 3d1f576cb47741deaf5390d56a3ed805
I20260926 07:05:45.856547 22702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:45.866693 22702 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:45.866869 22702 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:45.878064 22702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c380108fba594631a09ceb5572bfbaca: Generated new TSK 0
I20260926 07:05:45.878265 22702 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:45.880712 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22615
2026-09-26T07:05:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:45 dist-test-slave-t1mc krb5kdc[22714](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:45 dist-test-slave-t1mc krb5kdc[22714](info): set up 2 sockets
Sep 26 07:05:45 dist-test-slave-t1mc krb5kdc[22714](info): commencing operation
krb5kdc: starting...
W20260926 07:05:46.912868 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:46 dist-test-slave-t1mc krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:47.032707 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:47.114209 22727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:47.114472 22727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:47.114514 22727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:47.115640 22727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:47.115710 22727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:47.115734 22727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:47.115756 22727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:47.115785 22727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:47.117265 22727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:47.117548 22727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:47.117771 22727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:47.122699 22733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:47.122895 22727 server_base.cc:1034] running on GCE node
W20260926 07:05:47.122736 22732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:47.122704 22735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:47.123275 22727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:47.123615 22727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:47.124857 22727 hybrid_clock.cc:648] HybridClock initialized: now 1790406347124843 us; error 46 us; skew 500 ppm
Sep 26 07:05:47 dist-test-slave-t1mc krb5kdc[22714](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:47.127336 22727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:47.127815 22727 webserver.cc:460] Webserver started at http://127.0.0.1:38793/ using document root <none> and password file <none>
I20260926 07:05:47.128050 22727 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:47.128118 22727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:47.128252 22727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:47.129365 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "348ad3a8dcb7491497b1b6b98a509eee"
format_stamp: "Formatted at 2026-09-26 07:05:47 on dist-test-slave-t1mc"
server_key: "66452fcde0b18f6bf7a433c658ebc3e3"
server_key_iv: "6ba29a8b38474947d131fe91821bbad3"
server_key_version: "encryptionkey@0"
I20260926 07:05:47.129745 22727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "348ad3a8dcb7491497b1b6b98a509eee"
format_stamp: "Formatted at 2026-09-26 07:05:47 on dist-test-slave-t1mc"
server_key: "66452fcde0b18f6bf7a433c658ebc3e3"
server_key_iv: "6ba29a8b38474947d131fe91821bbad3"
server_key_version: "encryptionkey@0"
I20260926 07:05:47.131048 22727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 07:05:47.131770 22742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:47.131981 22727 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:05:47.132079 22727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "348ad3a8dcb7491497b1b6b98a509eee"
format_stamp: "Formatted at 2026-09-26 07:05:47 on dist-test-slave-t1mc"
server_key: "66452fcde0b18f6bf7a433c658ebc3e3"
server_key_iv: "6ba29a8b38474947d131fe91821bbad3"
server_key_version: "encryptionkey@0"
I20260926 07:05:47.132164 22727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:47.152976 22727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:47.154996 22727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:47.155165 22727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:47.159536 22727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33595
I20260926 07:05:47.159595 22794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33595 every 8 connection(s)
I20260926 07:05:47.159956 22727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:47.160920 22795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:47.163092 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: Bootstrap starting.
I20260926 07:05:47.163609 22795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:47.163848 22795 log.cc:824] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:47.164511 22795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: No bootstrap required, opened a new log
I20260926 07:05:47.166057 22795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260926 07:05:47.166190 22795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:47.166216 22795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348ad3a8dcb7491497b1b6b98a509eee, State: Initialized, Role: FOLLOWER
I20260926 07:05:47.166317 22795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260926 07:05:47.166407 22795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:47.166445 22795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:47.166494 22795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:47.167009 22795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260926 07:05:47.167129 22795 leader_election.cc:302] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348ad3a8dcb7491497b1b6b98a509eee; no voters:
I20260926 07:05:47.167306 22795 leader_election.cc:288] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:47.167371 22798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:47.167554 22798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [term 1 LEADER]: Becoming Leader. State: Replica: 348ad3a8dcb7491497b1b6b98a509eee, State: Running, Role: LEADER
I20260926 07:05:47.167562 22795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:47.167650 22798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260926 07:05:47.168051 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22727
I20260926 07:05:47.168210 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:47.168423 1893 external_mini_cluster.cc:1428] Setting key 4c6f05e7ca9ba541dd8e19ec72c1e9c9
I20260926 07:05:47.169085 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348ad3a8dcb7491497b1b6b98a509eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260926 07:05:47.169183 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348ad3a8dcb7491497b1b6b98a509eee. Latest consensus state: current_term: 1 leader_uuid: "348ad3a8dcb7491497b1b6b98a509eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348ad3a8dcb7491497b1b6b98a509eee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260926 07:05:47.169266 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:47.169288 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee [sys.catalog]: This master's current role is: LEADER
W20260926 07:05:47.169859 22815 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:47.169958 22815 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:47.170049 22816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:47.170200 22816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:47.171721 22816 catalog_manager.cc:1347] Generated new cluster ID: 8b01ca11b091494eb9720ec1858e3012
I20260926 07:05:47.171773 22816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:05:47 dist-test-slave-t1mc krb5kdc[22714](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:05:47.178990 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:47.168957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:53023 (local address 127.0.0.1:33595)
0926 07:05:47.169576 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:47.169585 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:47.169624 (+ 39us) server_negotiation.cc:407] Connection header received
0926 07:05:47.169739 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:47.169744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:47.169788 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:47.169943 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:47.170653 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:47.171980 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:47.172749 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:47.173239 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:47.173372 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:47.175706 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:47.175710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:47.175710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:47.175721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:47.177415 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:47.177808 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:47.177811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:47.177812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:47.177909 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:47.178136 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:47.178137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:47.178137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:47.178389 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:47.178596 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:47.178688 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:05:47.178795 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":106,"threads_started":1}
I20260926 07:05:47.179956 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:47.191063 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:47.181289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:44747 (local address 127.0.0.1:33595)
0926 07:05:47.181325 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:47.181327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:47.185021 (+ 3694us) server_negotiation.cc:407] Connection header received
0926 07:05:47.185104 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:47.185105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:47.185115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:47.185172 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:47.185737 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:47.186245 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:47.186979 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:47.187199 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:47.187275 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:47.188449 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:47.188451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:47.188451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:47.188458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:47.189887 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:47.190246 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:47.190247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:47.190247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:47.190300 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:47.190533 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:47.190534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:47.190534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:47.190645 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:47.190767 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:47.190885 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 07:05:47.190945 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:05:47.205430 22816 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:47.205981 22816 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:47.213762 22816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 348ad3a8dcb7491497b1b6b98a509eee: Generated new TSK 0
I20260926 07:05:47.213968 22816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:47.217967 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22727
W20260926 07:05:47.222424 22712 connection.cc:535] client connection to 127.0.0.1:33595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33595): Connection reset by peer (error 104)
2026-09-26T07:05:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:47 dist-test-slave-t1mc krb5kdc[22826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:47 dist-test-slave-t1mc krb5kdc[22826](info): set up 2 sockets
Sep 26 07:05:47 dist-test-slave-t1mc krb5kdc[22826](info): commencing operation
krb5kdc: starting...
W20260926 07:05:48.256914 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:48 dist-test-slave-t1mc krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:48.373473 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:48.457123 22839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:48.457314 22839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:48.457336 22839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:48.458462 22839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:48.458504 22839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:48.458518 22839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:48.458573 22839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:48.458595 22839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:48.460019 22839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:48.460260 22839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:48.460476 22839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:48.465216 22847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:48.465250 22845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:48.465212 22844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:48.465472 22839 server_base.cc:1034] running on GCE node
I20260926 07:05:48.465745 22839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:48.466025 22839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:48.467203 22839 hybrid_clock.cc:648] HybridClock initialized: now 1790406348467196 us; error 34 us; skew 500 ppm
Sep 26 07:05:48 dist-test-slave-t1mc krb5kdc[22826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:48.469597 22839 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:48.470021 22839 webserver.cc:460] Webserver started at http://127.0.0.1:33499/ using document root <none> and password file <none>
I20260926 07:05:48.470243 22839 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:48.470309 22839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:48.470459 22839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:48.471494 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "499a13386c8c42d994c70780321c8f63"
format_stamp: "Formatted at 2026-09-26 07:05:48 on dist-test-slave-t1mc"
server_key: "a4c958f8ec992f8ba8c15346cc390470"
server_key_iv: "b9ee3b633dc0ed7ed27aebda01fcce7e"
server_key_version: "encryptionkey@0"
I20260926 07:05:48.471854 22839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "499a13386c8c42d994c70780321c8f63"
format_stamp: "Formatted at 2026-09-26 07:05:48 on dist-test-slave-t1mc"
server_key: "a4c958f8ec992f8ba8c15346cc390470"
server_key_iv: "b9ee3b633dc0ed7ed27aebda01fcce7e"
server_key_version: "encryptionkey@0"
I20260926 07:05:48.473212 22839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:48.473930 22854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:48.474110 22839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 07:05:48.474211 22839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "499a13386c8c42d994c70780321c8f63"
format_stamp: "Formatted at 2026-09-26 07:05:48 on dist-test-slave-t1mc"
server_key: "a4c958f8ec992f8ba8c15346cc390470"
server_key_iv: "b9ee3b633dc0ed7ed27aebda01fcce7e"
server_key_version: "encryptionkey@0"
I20260926 07:05:48.474293 22839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:48.482492 22839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:48.484222 22839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:48.484396 22839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:48.488312 22839 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35883
I20260926 07:05:48.488359 22906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35883 every 8 connection(s)
I20260926 07:05:48.488709 22839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:48.489702 22907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:48.491906 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: Bootstrap starting.
I20260926 07:05:48.492483 22907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:48.492813 22907 log.cc:824] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:48.493508 22907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: No bootstrap required, opened a new log
I20260926 07:05:48.495046 22907 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260926 07:05:48.495193 22907 raft_consensus.cc:374] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:48.495215 22907 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499a13386c8c42d994c70780321c8f63, State: Initialized, Role: FOLLOWER
I20260926 07:05:48.495407 22907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260926 07:05:48.495469 22907 raft_consensus.cc:388] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:48.495489 22907 raft_consensus.cc:482] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:48.495554 22907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:48.496079 22907 raft_consensus.cc:504] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260926 07:05:48.496191 22907 leader_election.cc:302] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 499a13386c8c42d994c70780321c8f63; no voters:
I20260926 07:05:48.496371 22907 leader_election.cc:288] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:48.496456 22910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:48.496626 22910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [term 1 LEADER]: Becoming Leader. State: Replica: 499a13386c8c42d994c70780321c8f63, State: Running, Role: LEADER
I20260926 07:05:48.496646 22907 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:48.496714 22910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260926 07:05:48.497288 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "499a13386c8c42d994c70780321c8f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } } }
I20260926 07:05:48.497375 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:48.497493 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22839
I20260926 07:05:48.497529 22912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 499a13386c8c42d994c70780321c8f63. Latest consensus state: current_term: 1 leader_uuid: "499a13386c8c42d994c70780321c8f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499a13386c8c42d994c70780321c8f63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } } }
I20260926 07:05:48.497596 22912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:48.497609 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:48.497812 1893 external_mini_cluster.cc:1428] Setting key 8ee372d2c6b305a182eb796ce6132e5a
W20260926 07:05:48.498425 22926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:48.498525 22926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:48.498605 22919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:48.498790 22919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:48.500229 22919 catalog_manager.cc:1347] Generated new cluster ID: 9eaded22ace5493b8743c561bbb47609
I20260926 07:05:48.500278 22919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:48.504618 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:48.498348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:39995 (local address 127.0.0.1:35883)
0926 07:05:48.498528 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:48.498537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:48.498565 (+ 28us) server_negotiation.cc:407] Connection header received
0926 07:05:48.499334 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:48.499340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:48.499397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:48.499548 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:48.499778 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:48.501198 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:48.502276 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:48.502890 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:48.503003 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:48.503735 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:48.503739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:48.503739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:48.503945 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:48.503983 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:48.503997 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:48.504080 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:48.504172 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 07:05:48.504199 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260926 07:05:48.505649 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:48.507124 22919 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:48.507813 22919 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:48.510600 22929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:48.507082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:54443 (local address 127.0.0.1:35883)
0926 07:05:48.507116 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:48.507117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:48.507143 (+ 26us) server_negotiation.cc:407] Connection header received
0926 07:05:48.507235 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:48.507236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:48.507270 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:48.507313 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:48.507495 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:48.508177 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:48.509227 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:48.509534 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:48.509581 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:48.510273 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:48.510274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:48.510274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:48.510278 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:48.510290 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:48.510294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:48.510333 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:48.510518 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 07:05:48.510529 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:05:48.512225 22919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 499a13386c8c42d994c70780321c8f63: Generated new TSK 0
I20260926 07:05:48.512400 22919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:48.514865 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22839
W20260926 07:05:48.518647 22824 connection.cc:535] client connection to 127.0.0.1:35883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35883): Connection reset by peer (error 104)
2026-09-26T07:05:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:48 dist-test-slave-t1mc krb5kdc[22938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:48 dist-test-slave-t1mc krb5kdc[22938](info): set up 2 sockets
Sep 26 07:05:48 dist-test-slave-t1mc krb5kdc[22938](info): commencing operation
krb5kdc: starting...
W20260926 07:05:49.545141 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:49 dist-test-slave-t1mc krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:49.661701 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:49.742175 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:49.742413 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:49.742460 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:49.743688 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:49.743747 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:49.743773 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:49.743805 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:49.743834 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:49.745330 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:49.745615 22951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:49.745843 22951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:49.750610 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:49.750635 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:49.750710 22951 server_base.cc:1034] running on GCE node
W20260926 07:05:49.750717 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:49.751104 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:49.751412 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:49.752552 22951 hybrid_clock.cc:648] HybridClock initialized: now 1790406349752542 us; error 28 us; skew 500 ppm
Sep 26 07:05:49 dist-test-slave-t1mc krb5kdc[22938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:49.755087 22951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:49.755496 22951 webserver.cc:460] Webserver started at http://127.0.0.1:34829/ using document root <none> and password file <none>
I20260926 07:05:49.755787 22951 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:49.755856 22951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:49.756006 22951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:49.757126 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "e538f95adaa34e72bc7cf7adefdc1607"
format_stamp: "Formatted at 2026-09-26 07:05:49 on dist-test-slave-t1mc"
server_key: "70e9f1d274efa06522a6b367407c5105"
server_key_iv: "253ef5afeada8f99577401f7dbad3719"
server_key_version: "encryptionkey@0"
I20260926 07:05:49.757479 22951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "e538f95adaa34e72bc7cf7adefdc1607"
format_stamp: "Formatted at 2026-09-26 07:05:49 on dist-test-slave-t1mc"
server_key: "70e9f1d274efa06522a6b367407c5105"
server_key_iv: "253ef5afeada8f99577401f7dbad3719"
server_key_version: "encryptionkey@0"
I20260926 07:05:49.758809 22951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:49.759575 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:49.759753 22951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:49.759852 22951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "e538f95adaa34e72bc7cf7adefdc1607"
format_stamp: "Formatted at 2026-09-26 07:05:49 on dist-test-slave-t1mc"
server_key: "70e9f1d274efa06522a6b367407c5105"
server_key_iv: "253ef5afeada8f99577401f7dbad3719"
server_key_version: "encryptionkey@0"
I20260926 07:05:49.759933 22951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:49.763255 22951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:49.765625 22951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:49.765786 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:49.769480 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45043
I20260926 07:05:49.769529 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45043 every 8 connection(s)
I20260926 07:05:49.769834 22951 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:49.770816 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:49.773077 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607: Bootstrap starting.
I20260926 07:05:49.773612 23019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:49.773888 23019 log.cc:824] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:49.774552 23019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607: No bootstrap required, opened a new log
I20260926 07:05:49.776099 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 22951
I20260926 07:05:49.776065 23019 raft_consensus.cc:348] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } }
I20260926 07:05:49.776192 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:49.776227 23019 raft_consensus.cc:374] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:49.776292 23019 raft_consensus.cc:729] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e538f95adaa34e72bc7cf7adefdc1607, State: Initialized, Role: FOLLOWER
I20260926 07:05:49.776408 1893 external_mini_cluster.cc:1428] Setting key 5ac3dbf85ec58a4f088c994d6a567b2f
I20260926 07:05:49.776458 23019 consensus_queue.cc:260] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } }
I20260926 07:05:49.776539 23019 raft_consensus.cc:388] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:49.776597 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:49.776649 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:49.777711 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } }
I20260926 07:05:49.777828 23019 leader_election.cc:302] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e538f95adaa34e72bc7cf7adefdc1607; no voters:
I20260926 07:05:49.777997 23019 leader_election.cc:288] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:49.778035 23024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:49.778206 23024 raft_consensus.cc:686] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [term 1 LEADER]: Becoming Leader. State: Replica: e538f95adaa34e72bc7cf7adefdc1607, State: Running, Role: LEADER
I20260926 07:05:49.778311 23019 sys_catalog.cc:564] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:49.778328 23024 consensus_queue.cc:237] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } }
I20260926 07:05:49.778692 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:49.776908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:56385 (local address 127.0.0.1:45043)
0926 07:05:49.777070 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:49.777085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:49.777115 (+ 30us) server_negotiation.cc:407] Connection header received
0926 07:05:49.777241 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:49.777246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:49.777268 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:49.777373 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:49.778076 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:49.778080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:49.778080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:49.778092 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:49.778163 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:49.778197 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:49.778268 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:49.778473 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 07:05:49.778516 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260926 07:05:49.778808 23024 sys_catalog.cc:455] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e538f95adaa34e72bc7cf7adefdc1607. Latest consensus state: current_term: 1 leader_uuid: "e538f95adaa34e72bc7cf7adefdc1607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } } }
I20260926 07:05:49.778939 23025 sys_catalog.cc:455] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e538f95adaa34e72bc7cf7adefdc1607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e538f95adaa34e72bc7cf7adefdc1607" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45043 } } }
I20260926 07:05:49.778970 23024 sys_catalog.cc:458] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:49.779085 23025 sys_catalog.cc:458] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:49.779578 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:49.779888 23033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:49.780371 23033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:49.782091 23033 catalog_manager.cc:1347] Generated new cluster ID: 574092f0ee834b40a2b8ed7dd95fc4be
I20260926 07:05:49.782137 23033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:49.782445 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:49.781405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:33185 (local address 127.0.0.1:45043)
0926 07:05:49.781438 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:49.781440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:49.781448 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:05:49.781527 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:49.781528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:49.781533 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:49.781592 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:49.782206 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:49.782207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:49.782208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:49.782212 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:49.782224 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:49.782228 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:49.782285 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:49.782342 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 07:05:49.782363 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:05:49.790127 23033 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:49.790282 23033 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:49.797001 23033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e538f95adaa34e72bc7cf7adefdc1607: Generated new TSK 0
I20260926 07:05:49.797192 23033 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:49.803010 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 22951
2026-09-26T07:05:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1288 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:49 dist-test-slave-t1mc krb5kdc[23050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:49 dist-test-slave-t1mc krb5kdc[23050](info): set up 2 sockets
Sep 26 07:05:49 dist-test-slave-t1mc krb5kdc[23050](info): commencing operation
krb5kdc: starting...
W20260926 07:05:50.836931 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:50 dist-test-slave-t1mc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:50.955700 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:51.034924 23063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:51.035173 23063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:51.035211 23063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:51.036361 23063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:51.036432 23063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:51.036468 23063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:51.036500 23063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:51.036523 23063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:51.038015 23063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38643
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:51.038295 23063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:51.038527 23063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:51.043525 23068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:51.043659 23063 server_base.cc:1034] running on GCE node
W20260926 07:05:51.043596 23069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:51.043637 23071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:51.044003 23063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:51.044330 23063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:51.045501 23063 hybrid_clock.cc:648] HybridClock initialized: now 1790406351045489 us; error 33 us; skew 500 ppm
Sep 26 07:05:51 dist-test-slave-t1mc krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:51.051251 23063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:51.051733 23063 webserver.cc:460] Webserver started at http://127.0.0.1:39003/ using document root <none> and password file <none>
I20260926 07:05:51.051996 23063 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:51.052044 23063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:51.052218 23063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:51.053366 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d42e65c46cbc47af834dcffb8f17f6c3"
format_stamp: "Formatted at 2026-09-26 07:05:51 on dist-test-slave-t1mc"
server_key: "3f6f920b3f57a5fba251c3c655a8db01"
server_key_iv: "2aea8254142b181011db491b9eba186f"
server_key_version: "encryptionkey@0"
I20260926 07:05:51.053730 23063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d42e65c46cbc47af834dcffb8f17f6c3"
format_stamp: "Formatted at 2026-09-26 07:05:51 on dist-test-slave-t1mc"
server_key: "3f6f920b3f57a5fba251c3c655a8db01"
server_key_iv: "2aea8254142b181011db491b9eba186f"
server_key_version: "encryptionkey@0"
I20260926 07:05:51.055125 23063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:51.055891 23078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:51.056087 23063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:51.056185 23063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d42e65c46cbc47af834dcffb8f17f6c3"
format_stamp: "Formatted at 2026-09-26 07:05:51 on dist-test-slave-t1mc"
server_key: "3f6f920b3f57a5fba251c3c655a8db01"
server_key_iv: "2aea8254142b181011db491b9eba186f"
server_key_version: "encryptionkey@0"
I20260926 07:05:51.056267 23063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:51.066431 23063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:51.068451 23063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:51.068609 23063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:51.072530 23063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38643
I20260926 07:05:51.072598 23130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38643 every 8 connection(s)
I20260926 07:05:51.073009 23063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:51.074023 23131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:51.076277 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3: Bootstrap starting.
I20260926 07:05:51.076934 23131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:51.077225 23131 log.cc:824] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:51.077945 23131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3: No bootstrap required, opened a new log
I20260926 07:05:51.079540 23131 raft_consensus.cc:348] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } }
I20260926 07:05:51.079676 23131 raft_consensus.cc:374] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:51.079699 23131 raft_consensus.cc:729] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d42e65c46cbc47af834dcffb8f17f6c3, State: Initialized, Role: FOLLOWER
I20260926 07:05:51.079883 23131 consensus_queue.cc:260] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } }
I20260926 07:05:51.079974 23131 raft_consensus.cc:388] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:51.079998 23131 raft_consensus.cc:482] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:51.080058 23131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:51.080157 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23063
I20260926 07:05:51.080296 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:51.080533 1893 external_mini_cluster.cc:1428] Setting key 1545b821157d8fd1887be9ec7f82f12b
I20260926 07:05:51.080760 23131 raft_consensus.cc:504] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } }
I20260926 07:05:51.080902 23131 leader_election.cc:302] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d42e65c46cbc47af834dcffb8f17f6c3; no voters:
I20260926 07:05:51.081075 23131 leader_election.cc:288] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:51.081250 23136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:51.081360 23131 sys_catalog.cc:564] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:51.081372 23136 raft_consensus.cc:686] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [term 1 LEADER]: Becoming Leader. State: Replica: d42e65c46cbc47af834dcffb8f17f6c3, State: Running, Role: LEADER
I20260926 07:05:51.081532 23136 consensus_queue.cc:237] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } }
I20260926 07:05:51.082080 23135 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } } }
I20260926 07:05:51.082171 23135 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:51.082183 23138 sys_catalog.cc:455] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d42e65c46cbc47af834dcffb8f17f6c3. Latest consensus state: current_term: 1 leader_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d42e65c46cbc47af834dcffb8f17f6c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38643 } } }
I20260926 07:05:51.082275 23138 sys_catalog.cc:458] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:51.082489 23145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:51.083091 23145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:51.084677 23145 catalog_manager.cc:1347] Generated new cluster ID: ae9fd8cf90e341a8b5396b3f39422bb8
I20260926 07:05:51.085520 23145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 07:05:51 dist-test-slave-t1mc krb5kdc[23050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 07:05:51.091861 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:51.081099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:45969 (local address 127.0.0.1:38643)
0926 07:05:51.081576 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:51.081586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:51.081621 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:05:51.081748 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:51.081754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:51.081801 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:51.081996 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:51.082807 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:51.084289 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:51.085505 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:51.085949 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:51.086087 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:51.088445 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:51.088449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:51.088449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:51.088462 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:51.090200 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:51.090603 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:51.090606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:51.090606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:51.090709 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:51.090945 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:51.090946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:51.090946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:51.091201 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:51.091393 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:51.091489 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 07:05:51.091695 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":97,"threads_started":1}
I20260926 07:05:51.092898 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:51.101434 23145 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:51.102144 23145 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:51.107376 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:51.094283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:44327 (local address 127.0.0.1:38643)
0926 07:05:51.094319 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:51.094321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:51.101095 (+ 6774us) server_negotiation.cc:407] Connection header received
0926 07:05:51.101148 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:51.101149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:51.101162 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:51.101229 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:51.101878 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:51.102491 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:51.103197 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:51.103414 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:51.103518 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:51.104748 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:51.104749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:51.104749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:51.104758 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:51.106207 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:51.106575 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:51.106576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:51.106576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:51.106635 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:51.106856 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:51.106857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:51.106858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:51.106967 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:51.107091 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:51.107202 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 07:05:51.107263 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":18}
I20260926 07:05:51.111179 23145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d42e65c46cbc47af834dcffb8f17f6c3: Generated new TSK 0
I20260926 07:05:51.111354 23145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:51.114114 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23063
W20260926 07:05:51.118336 23048 connection.cc:535] client connection to 127.0.0.1:38643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38643): Connection reset by peer (error 104)
2026-09-26T07:05:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:51 dist-test-slave-t1mc krb5kdc[23162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:51 dist-test-slave-t1mc krb5kdc[23162](info): set up 2 sockets
Sep 26 07:05:51 dist-test-slave-t1mc krb5kdc[23162](info): commencing operation
krb5kdc: starting...
W20260926 07:05:52.144833 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:52 dist-test-slave-t1mc krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:52.267138 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:52.346776 23175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:52.347002 23175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:52.347041 23175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:52.348193 23175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:52.348261 23175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:52.348285 23175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:52.348306 23175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:52.348336 23175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:52.349947 23175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:52.350234 23175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:52.350454 23175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:52.355185 23183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:52.355222 23175 server_base.cc:1034] running on GCE node
W20260926 07:05:52.355191 23181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:52.355170 23180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:52.355748 23175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:52.356045 23175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:52.357216 23175 hybrid_clock.cc:648] HybridClock initialized: now 1790406352357212 us; error 45 us; skew 500 ppm
Sep 26 07:05:52 dist-test-slave-t1mc krb5kdc[23162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:52.359670 23175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:52.360116 23175 webserver.cc:460] Webserver started at http://127.0.0.1:40329/ using document root <none> and password file <none>
I20260926 07:05:52.360337 23175 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:52.360416 23175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:52.360564 23175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:52.361694 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "524a0b5d9084494e9a63f378bdb2e826"
format_stamp: "Formatted at 2026-09-26 07:05:52 on dist-test-slave-t1mc"
server_key: "736844bf2de6a99142308f3101cae97b"
server_key_iv: "c01c9faac61d8f6291bd61f05b69b9f5"
server_key_version: "encryptionkey@0"
I20260926 07:05:52.362061 23175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "524a0b5d9084494e9a63f378bdb2e826"
format_stamp: "Formatted at 2026-09-26 07:05:52 on dist-test-slave-t1mc"
server_key: "736844bf2de6a99142308f3101cae97b"
server_key_iv: "c01c9faac61d8f6291bd61f05b69b9f5"
server_key_version: "encryptionkey@0"
I20260926 07:05:52.363348 23175 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:52.364224 23190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:52.364449 23175 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:52.364557 23175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "524a0b5d9084494e9a63f378bdb2e826"
format_stamp: "Formatted at 2026-09-26 07:05:52 on dist-test-slave-t1mc"
server_key: "736844bf2de6a99142308f3101cae97b"
server_key_iv: "c01c9faac61d8f6291bd61f05b69b9f5"
server_key_version: "encryptionkey@0"
I20260926 07:05:52.364643 23175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:52.398850 23175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:52.400738 23175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:52.400944 23175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:52.404647 23175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37273
I20260926 07:05:52.404695 23242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37273 every 8 connection(s)
I20260926 07:05:52.405085 23175 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:52.405942 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:52.408200 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826: Bootstrap starting.
I20260926 07:05:52.408839 23243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:52.409124 23243 log.cc:824] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:52.409811 23243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826: No bootstrap required, opened a new log
I20260926 07:05:52.411345 23243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260926 07:05:52.411477 23243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:52.411499 23243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 524a0b5d9084494e9a63f378bdb2e826, State: Initialized, Role: FOLLOWER
I20260926 07:05:52.411661 23243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260926 07:05:52.411721 23243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:52.411741 23243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:52.411808 23243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:52.412070 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23175
I20260926 07:05:52.412205 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:52.412401 1893 external_mini_cluster.cc:1428] Setting key 59426e9507cc83bb681aa51b2be0c351
I20260926 07:05:52.412432 23243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260926 07:05:52.412556 23243 leader_election.cc:302] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 524a0b5d9084494e9a63f378bdb2e826; no voters:
I20260926 07:05:52.412737 23243 leader_election.cc:288] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:52.412781 23246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:52.413051 23246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [term 1 LEADER]: Becoming Leader. State: Replica: 524a0b5d9084494e9a63f378bdb2e826, State: Running, Role: LEADER
I20260926 07:05:52.413082 23243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:52.413213 23246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260926 07:05:52.413810 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "524a0b5d9084494e9a63f378bdb2e826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260926 07:05:52.413902 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:52.414057 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 524a0b5d9084494e9a63f378bdb2e826. Latest consensus state: current_term: 1 leader_uuid: "524a0b5d9084494e9a63f378bdb2e826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "524a0b5d9084494e9a63f378bdb2e826" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260926 07:05:52.414113 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:52.414127 23257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:52.414582 23257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:52.416106 23257 catalog_manager.cc:1347] Generated new cluster ID: 700571dbb38b45d78835e2db755411d0
I20260926 07:05:52.416158 23257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:52.418609 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:52.412909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:56271 (local address 127.0.0.1:37273)
0926 07:05:52.413348 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:52.413357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:52.413393 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:05:52.413517 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:52.413525 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:52.413587 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:52.413729 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:52.413943 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:52.415577 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:52.416506 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:52.417110 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:52.417205 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:52.417932 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:52.417936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:52.417936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:52.418129 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:52.418164 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:52.418178 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:52.418255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:52.418376 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 07:05:52.418400 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":61,"threads_started":1}
I20260926 07:05:52.419697 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:52.424642 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:52.421271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:44493 (local address 127.0.0.1:37273)
0926 07:05:52.421304 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:52.421305 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:52.421313 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:05:52.421381 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:52.421382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:52.421391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:52.421450 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:52.421582 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:52.422275 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:52.423333 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:52.423647 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:52.423706 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:52.424372 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:52.424373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:52.424373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:52.424377 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:52.424390 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:52.424395 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:52.424473 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:52.424558 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 07:05:52.424570 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:05:52.426164 23257 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:52.426827 23257 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:52.435271 23257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 524a0b5d9084494e9a63f378bdb2e826: Generated new TSK 0
I20260926 07:05:52.435446 23257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:52.441169 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23175
W20260926 07:05:52.445354 23160 connection.cc:535] client connection to 127.0.0.1:37273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37273): Connection reset by peer (error 104)
2026-09-26T07:05:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:52 dist-test-slave-t1mc krb5kdc[23274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:52 dist-test-slave-t1mc krb5kdc[23274](info): set up 2 sockets
Sep 26 07:05:52 dist-test-slave-t1mc krb5kdc[23274](info): commencing operation
krb5kdc: starting...
W20260926 07:05:53.480953 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:53 dist-test-slave-t1mc krb5kdc[23274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 07:05:53.600076 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:53.679276 23287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:53.679508 23287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:53.679546 23287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:53.680672 23287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:53.680748 23287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:53.680771 23287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:53.680826 23287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:53.680855 23287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:53.682293 23287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:53.682562 23287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:53.682806 23287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:53.687824 23292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:53.687881 23287 server_base.cc:1034] running on GCE node
W20260926 07:05:53.687849 23295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:53.687849 23293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:53.688271 23287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:53.688508 23287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:53.689651 23287 hybrid_clock.cc:648] HybridClock initialized: now 1790406353689637 us; error 28 us; skew 500 ppm
Sep 26 07:05:53 dist-test-slave-t1mc krb5kdc[23274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:53.692239 23287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 07:05:53.692672 23287 webserver.cc:460] Webserver started at http://127.0.0.1:44905/ using document root <none> and password file <none>
I20260926 07:05:53.692938 23287 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:53.693007 23287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:53.693171 23287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:53.694213 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "9e7bb0c1d15e4b608eb17d9a642ad012"
format_stamp: "Formatted at 2026-09-26 07:05:53 on dist-test-slave-t1mc"
server_key: "947c8b526ba4124c9212c65791fc841a"
server_key_iv: "b434cc489adc202d435e0edc6dc812a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:53.694593 23287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "9e7bb0c1d15e4b608eb17d9a642ad012"
format_stamp: "Formatted at 2026-09-26 07:05:53 on dist-test-slave-t1mc"
server_key: "947c8b526ba4124c9212c65791fc841a"
server_key_iv: "b434cc489adc202d435e0edc6dc812a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:53.695863 23287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:05:53.696551 23302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:53.696722 23287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:53.696836 23287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "9e7bb0c1d15e4b608eb17d9a642ad012"
format_stamp: "Formatted at 2026-09-26 07:05:53 on dist-test-slave-t1mc"
server_key: "947c8b526ba4124c9212c65791fc841a"
server_key_iv: "b434cc489adc202d435e0edc6dc812a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:53.696920 23287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:53.700083 23287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:53.702368 23287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:53.702521 23287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:53.706418 23287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260926 07:05:53.706449 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260926 07:05:53.706756 23287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:53.707760 23355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:53.709997 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: Bootstrap starting.
I20260926 07:05:53.710556 23355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:53.710835 23355 log.cc:824] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:53.711508 23355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: No bootstrap required, opened a new log
I20260926 07:05:53.713132 23355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260926 07:05:53.713263 23355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:53.713285 23355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e7bb0c1d15e4b608eb17d9a642ad012, State: Initialized, Role: FOLLOWER
I20260926 07:05:53.713490 23355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260926 07:05:53.713573 23355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:53.713622 23355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:53.713672 23355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:53.714185 23355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260926 07:05:53.714298 23355 leader_election.cc:302] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e7bb0c1d15e4b608eb17d9a642ad012; no voters:
I20260926 07:05:53.714478 23355 leader_election.cc:288] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:53.714581 23358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:53.714676 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23287
I20260926 07:05:53.714731 23355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:53.714802 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:53.714798 23358 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [term 1 LEADER]: Becoming Leader. State: Replica: 9e7bb0c1d15e4b608eb17d9a642ad012, State: Running, Role: LEADER
I20260926 07:05:53.714881 23358 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260926 07:05:53.715008 1893 external_mini_cluster.cc:1428] Setting key be56a178418e3866b838ec7dbbd6ae30
I20260926 07:05:53.716195 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e7bb0c1d15e4b608eb17d9a642ad012. Latest consensus state: current_term: 1 leader_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260926 07:05:53.716297 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:53.716527 23359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e7bb0c1d15e4b608eb17d9a642ad012" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260926 07:05:53.716593 23359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012 [sys.catalog]: This master's current role is: LEADER
W20260926 07:05:53.716625 23374 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:05:53.716720 23374 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:05:53.716812 23375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:53.717005 23375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:53.717795 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:53.715630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:50219 (local address 127.0.0.1:39121)
0926 07:05:53.715885 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:53.715895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:53.715927 (+ 32us) server_negotiation.cc:407] Connection header received
0926 07:05:53.716042 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:53.716050 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:53.716083 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:53.717040 (+ 957us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:53.717341 (+ 301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:53.717344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:53.717345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:53.717348 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:53.717380 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:53.717395 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:53.717467 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:53.717602 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 07:05:53.717650 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260926 07:05:53.718559 23375 catalog_manager.cc:1347] Generated new cluster ID: d8ddef2b39474de4a36644a43444286e
I20260926 07:05:53.718605 23375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:53.720954 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:53.723505 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:53.722392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.242:43869 (local address 127.0.0.1:39121)
0926 07:05:53.722430 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:53.722432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:53.722564 (+ 132us) server_negotiation.cc:407] Connection header received
0926 07:05:53.722592 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:53.722593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:53.722598 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:53.722666 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:53.723278 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:53.723279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:53.723279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:53.723283 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:53.723299 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:53.723304 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:53.723365 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:53.723418 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 07:05:53.723432 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 07:05:53.726325 23375 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:53.726498 23375 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:53.728577 23375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e7bb0c1d15e4b608eb17d9a642ad012: Generated new TSK 0
I20260926 07:05:53.728753 23375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:53.735360 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23287
2026-09-26T07:05:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:53 dist-test-slave-t1mc krb5kdc[23386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:53 dist-test-slave-t1mc krb5kdc[23386](info): set up 2 sockets
Sep 26 07:05:53 dist-test-slave-t1mc krb5kdc[23386](info): commencing operation
krb5kdc: starting...
W20260926 07:05:54.757022 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:54 dist-test-slave-t1mc krb5kdc[23386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:54.871865 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42817
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:54.952329 23399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:54.952581 23399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:54.952626 23399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:54.953892 23399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:54.953969 23399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:54.953997 23399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:54.954030 23399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:54.954056 23399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:54.955557 23399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:54.955874 23399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:54.956122 23399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:54.960942 23404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:54.960920 23405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:54.961027 23399 server_base.cc:1034] running on GCE node
W20260926 07:05:54.960920 23407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:54.961354 23399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:54.961695 23399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:54.962843 23399 hybrid_clock.cc:648] HybridClock initialized: now 1790406354962833 us; error 28 us; skew 500 ppm
Sep 26 07:05:54 dist-test-slave-t1mc krb5kdc[23386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406354, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:54.965360 23399 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:54.965762 23399 webserver.cc:460] Webserver started at http://127.1.217.126:41881/ using document root <none> and password file <none>
I20260926 07:05:54.965986 23399 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:54.966030 23399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:54.966167 23399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:54.967206 23399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "c75a70e78f4847d089bd9504fff366ad"
format_stamp: "Formatted at 2026-09-26 07:05:54 on dist-test-slave-t1mc"
server_key: "9fac3bbe9df6b46b81812d0b8468c118"
server_key_iv: "1cf5d0387fc66c5f3d2647e53d413d57"
server_key_version: "encryptionkey@0"
I20260926 07:05:54.967571 23399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "c75a70e78f4847d089bd9504fff366ad"
format_stamp: "Formatted at 2026-09-26 07:05:54 on dist-test-slave-t1mc"
server_key: "9fac3bbe9df6b46b81812d0b8468c118"
server_key_iv: "1cf5d0387fc66c5f3d2647e53d413d57"
server_key_version: "encryptionkey@0"
I20260926 07:05:54.968889 23399 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:05:54.969630 23414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:54.969815 23399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:54.969905 23399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "c75a70e78f4847d089bd9504fff366ad"
format_stamp: "Formatted at 2026-09-26 07:05:54 on dist-test-slave-t1mc"
server_key: "9fac3bbe9df6b46b81812d0b8468c118"
server_key_iv: "1cf5d0387fc66c5f3d2647e53d413d57"
server_key_version: "encryptionkey@0"
I20260926 07:05:54.969980 23399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:54.978590 23399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:54.980373 23399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:54.980537 23399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:54.984383 23399 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42817
I20260926 07:05:54.984419 23466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42817 every 8 connection(s)
I20260926 07:05:54.984840 23399 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:54.985724 23467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:54.986172 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23399
I20260926 07:05:54.986315 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:54.986490 1893 external_mini_cluster.cc:1428] Setting key b5861194b7dc9e41abab0721ae42eb32
I20260926 07:05:54.988361 23467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad: Bootstrap starting.
I20260926 07:05:54.988993 23467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:54.989286 23467 log.cc:824] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:54.989943 23467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad: No bootstrap required, opened a new log
I20260926 07:05:54.991873 23467 raft_consensus.cc:348] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } }
I20260926 07:05:54.992031 23467 raft_consensus.cc:374] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:54.992107 23467 raft_consensus.cc:729] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75a70e78f4847d089bd9504fff366ad, State: Initialized, Role: FOLLOWER
I20260926 07:05:54.992282 23467 consensus_queue.cc:260] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } }
I20260926 07:05:54.992383 23467 raft_consensus.cc:388] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:54.992429 23467 raft_consensus.cc:482] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:54.992477 23467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 0 FOLLOWER]: Advancing to term 1
Sep 26 07:05:54 dist-test-slave-t1mc krb5kdc[23386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:54.993094 23467 raft_consensus.cc:504] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } }
I20260926 07:05:54.993206 23467 leader_election.cc:302] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c75a70e78f4847d089bd9504fff366ad; no voters:
I20260926 07:05:54.993389 23467 leader_election.cc:288] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:54.993422 23472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:54.993537 23472 raft_consensus.cc:686] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [term 1 LEADER]: Becoming Leader. State: Replica: c75a70e78f4847d089bd9504fff366ad, State: Running, Role: LEADER
I20260926 07:05:54.993659 23467 sys_catalog.cc:564] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:54.993669 23472 consensus_queue.cc:237] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } }
I20260926 07:05:54.994207 23474 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c75a70e78f4847d089bd9504fff366ad. Latest consensus state: current_term: 1 leader_uuid: "c75a70e78f4847d089bd9504fff366ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } } }
I20260926 07:05:54.994346 23474 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:54.994588 23473 sys_catalog.cc:455] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c75a70e78f4847d089bd9504fff366ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c75a70e78f4847d089bd9504fff366ad" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42817 } } }
I20260926 07:05:54.994612 23481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:54.994701 23473 sys_catalog.cc:458] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:54.995191 23481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:54.996724 23481 catalog_manager.cc:1347] Generated new cluster ID: 5159152fda40435ca289c47bdae646bb
I20260926 07:05:54.996824 23481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:54.997773 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:54.987043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38021 (local address 127.1.217.126:42817)
0926 07:05:54.987236 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:54.987245 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:54.987280 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:05:54.987675 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:54.987681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:54.987725 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:54.987890 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:54.988543 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:54.989638 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:54.990452 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:54.990876 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:54.991006 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:54.993654 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:54.993658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:54.993659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:54.993671 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:54.996148 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:54.996556 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:54.996559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:54.996560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:54.996692 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:54.996961 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:54.996962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:54.996962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:54.997179 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:54.997358 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:54.997483 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 07:05:54.997589 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260926 07:05:54.998732 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:55.006410 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:55.000181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33293 (local address 127.1.217.126:42817)
0926 07:05:55.000214 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:55.000216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:55.000397 (+ 181us) server_negotiation.cc:407] Connection header received
0926 07:05:55.000454 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:55.000455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:55.000464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:55.000518 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:55.001135 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:55.001610 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:55.002309 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:55.002543 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:55.002629 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:55.003809 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:55.003810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:55.003811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:55.003818 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:55.005258 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:55.005620 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:55.005622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:55.005622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:55.005675 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:55.005907 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:55.005908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:55.005909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:55.006020 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:55.006130 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:55.006254 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:05:55.006314 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 07:05:55.021118 23481 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:55.021684 23481 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:55.024989 23481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c75a70e78f4847d089bd9504fff366ad: Generated new TSK 0
I20260926 07:05:55.025204 23481 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:55.027669 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23399
W20260926 07:05:55.031869 23384 connection.cc:535] client connection to 127.1.217.126:42817 recv error: Network error: failed to read from TLS socket (remote: 127.1.217.126:42817): Connection reset by peer (error 104)
2026-09-26T07:05:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:55 dist-test-slave-t1mc krb5kdc[23498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:55 dist-test-slave-t1mc krb5kdc[23498](info): set up 2 sockets
Sep 26 07:05:55 dist-test-slave-t1mc krb5kdc[23498](info): commencing operation
krb5kdc: starting...
W20260926 07:05:56.064757 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:56 dist-test-slave-t1mc krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:56.179943 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34453
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:56.262199 23511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:56.262395 23511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:56.262418 23511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:56.263561 23511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:56.263641 23511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:56.263667 23511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:56.263700 23511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:56.263725 23511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:56.265241 23511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:56.265502 23511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:56.265727 23511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:56.270401 23517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:56.270394 23516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:56.270419 23511 server_base.cc:1034] running on GCE node
W20260926 07:05:56.270394 23519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:56.270812 23511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:56.271126 23511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:56.272270 23511 hybrid_clock.cc:648] HybridClock initialized: now 1790406356272262 us; error 30 us; skew 500 ppm
Sep 26 07:05:56 dist-test-slave-t1mc krb5kdc[23498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406356, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:56.287052 23511 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:56.287580 23511 webserver.cc:460] Webserver started at http://127.1.217.126:39115/ using document root <none> and password file <none>
I20260926 07:05:56.287783 23511 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:56.287824 23511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:56.287915 23511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:56.289011 23511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "d8c27772a63e411fa3ecdda69a7fb17d"
format_stamp: "Formatted at 2026-09-26 07:05:56 on dist-test-slave-t1mc"
server_key: "e4c2db6ea81ee661ab078c2fab714199"
server_key_iv: "ec5c57d531f15a41b962e730aebf193c"
server_key_version: "encryptionkey@0"
I20260926 07:05:56.289318 23511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "d8c27772a63e411fa3ecdda69a7fb17d"
format_stamp: "Formatted at 2026-09-26 07:05:56 on dist-test-slave-t1mc"
server_key: "e4c2db6ea81ee661ab078c2fab714199"
server_key_iv: "ec5c57d531f15a41b962e730aebf193c"
server_key_version: "encryptionkey@0"
I20260926 07:05:56.290571 23511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:56.291296 23526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:56.291493 23511 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:05:56.291584 23511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "d8c27772a63e411fa3ecdda69a7fb17d"
format_stamp: "Formatted at 2026-09-26 07:05:56 on dist-test-slave-t1mc"
server_key: "e4c2db6ea81ee661ab078c2fab714199"
server_key_iv: "ec5c57d531f15a41b962e730aebf193c"
server_key_version: "encryptionkey@0"
I20260926 07:05:56.291639 23511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:56.300369 23511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:56.302105 23511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:56.302208 23511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:56.305904 23511 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34453
I20260926 07:05:56.305961 23578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34453 every 8 connection(s)
I20260926 07:05:56.306303 23511 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:56.307153 23579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:56.309392 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d: Bootstrap starting.
I20260926 07:05:56.309984 23579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:56.310277 23579 log.cc:824] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:56.310984 23579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d: No bootstrap required, opened a new log
I20260926 07:05:56.312543 23579 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } }
I20260926 07:05:56.312681 23579 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:56.312704 23579 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c27772a63e411fa3ecdda69a7fb17d, State: Initialized, Role: FOLLOWER
I20260926 07:05:56.312920 23579 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } }
I20260926 07:05:56.313014 23579 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:56.313037 23579 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:56.313067 23579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:56.313910 23579 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } }
I20260926 07:05:56.314031 23579 leader_election.cc:302] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c27772a63e411fa3ecdda69a7fb17d; no voters:
I20260926 07:05:56.314229 23579 leader_election.cc:288] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:56.314308 23582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:56.314396 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23511
I20260926 07:05:56.314487 23582 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [term 1 LEADER]: Becoming Leader. State: Replica: d8c27772a63e411fa3ecdda69a7fb17d, State: Running, Role: LEADER
I20260926 07:05:56.314509 23579 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:56.314519 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:56.314572 23582 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } }
I20260926 07:05:56.314752 1893 external_mini_cluster.cc:1428] Setting key cee8f1448234cc4b812da605815b6bb3
I20260926 07:05:56.315011 23583 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } } }
I20260926 07:05:56.315093 23583 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:56.315353 23584 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c27772a63e411fa3ecdda69a7fb17d. Latest consensus state: current_term: 1 leader_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c27772a63e411fa3ecdda69a7fb17d" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34453 } } }
I20260926 07:05:56.315466 23584 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:56.315629 23591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:56.316186 23591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:56.318048 23591 catalog_manager.cc:1347] Generated new cluster ID: 0d77ce5890914b74abb9ebd4664efb59
I20260926 07:05:56.318104 23591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:56.321467 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:56.315448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40823 (local address 127.1.217.126:34453)
0926 07:05:56.315887 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:56.315897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:56.315931 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:05:56.316052 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:56.316060 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:56.316116 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:56.316253 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:56.316499 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:56.317764 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:56.318890 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:56.319494 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:56.319882 (+ 388us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:56.320571 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:56.320574 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:56.320574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:56.320787 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:56.320824 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:56.320838 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:56.321122 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:56.321169 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:05:56.321201 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":82,"threads_started":1}
I20260926 07:05:56.322537 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:56.327873 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:56.324034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41253 (local address 127.1.217.126:34453)
0926 07:05:56.324084 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:56.324085 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:56.324306 (+ 221us) server_negotiation.cc:407] Connection header received
0926 07:05:56.324378 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:56.324379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:56.324398 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:56.324448 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:56.324601 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:56.325345 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:56.326380 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:56.326701 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:56.326755 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:56.327591 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:56.327592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:56.327592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:56.327596 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:56.327610 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:56.327614 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:56.327673 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:56.327787 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 07:05:56.327798 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:05:56.329052 23591 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:56.329754 23591 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:56.340096 23591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c27772a63e411fa3ecdda69a7fb17d: Generated new TSK 0
I20260926 07:05:56.340291 23591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:56.349535 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23511
W20260926 07:05:56.353317 23496 connection.cc:535] client connection to 127.1.217.126:34453 recv error: Network error: failed to read from TLS socket (remote: 127.1.217.126:34453): Connection reset by peer (error 104)
2026-09-26T07:05:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:56 dist-test-slave-t1mc krb5kdc[23610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:56 dist-test-slave-t1mc krb5kdc[23610](info): set up 2 sockets
Sep 26 07:05:56 dist-test-slave-t1mc krb5kdc[23610](info): commencing operation
krb5kdc: starting...
W20260926 07:05:57.380903 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:57 dist-test-slave-t1mc krb5kdc[23610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:57.499750 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34781
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:40151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:57.579689 23623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:57.579933 23623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:57.579974 23623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:57.581148 23623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:57.581219 23623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:57.581245 23623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:57.581279 23623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:57.581306 23623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:57.582769 23623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:40151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:57.583057 23623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:57.583294 23623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:05:57.588276 23628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:57.588295 23623 server_base.cc:1034] running on GCE node
W20260926 07:05:57.588276 23631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:57.588305 23629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:57.588876 23623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:57.589135 23623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:57.590281 23623 hybrid_clock.cc:648] HybridClock initialized: now 1790406357590266 us; error 30 us; skew 500 ppm
Sep 26 07:05:57 dist-test-slave-t1mc krb5kdc[23610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406357, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:57.592644 23623 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:57.593142 23623 webserver.cc:460] Webserver started at http://127.1.217.126:44617/ using document root <none> and password file <none>
I20260926 07:05:57.593367 23623 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:57.593434 23623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:57.593583 23623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:57.594712 23623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "e69c423c0dd94134b8ec3970b294dccb"
format_stamp: "Formatted at 2026-09-26 07:05:57 on dist-test-slave-t1mc"
server_key: "ff5bd7f081f44f21fd358a288b3d9685"
server_key_iv: "b499b8de880d45f7de0a60daf432de5c"
server_key_version: "encryptionkey@0"
I20260926 07:05:57.595095 23623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "e69c423c0dd94134b8ec3970b294dccb"
format_stamp: "Formatted at 2026-09-26 07:05:57 on dist-test-slave-t1mc"
server_key: "ff5bd7f081f44f21fd358a288b3d9685"
server_key_iv: "b499b8de880d45f7de0a60daf432de5c"
server_key_version: "encryptionkey@0"
I20260926 07:05:57.596392 23623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 07:05:57.597167 23638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:57.597340 23623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:57.597438 23623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "e69c423c0dd94134b8ec3970b294dccb"
format_stamp: "Formatted at 2026-09-26 07:05:57 on dist-test-slave-t1mc"
server_key: "ff5bd7f081f44f21fd358a288b3d9685"
server_key_iv: "b499b8de880d45f7de0a60daf432de5c"
server_key_version: "encryptionkey@0"
I20260926 07:05:57.597520 23623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:57.600744 23623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:57.602589 23623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:57.602756 23623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:57.607062 23623 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34781
I20260926 07:05:57.607090 23690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34781 every 8 connection(s)
I20260926 07:05:57.607432 23623 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:57.608392 23691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:57.610728 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb: Bootstrap starting.
I20260926 07:05:57.611304 23691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:57.611590 23691 log.cc:824] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:57.612259 23691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb: No bootstrap required, opened a new log
I20260926 07:05:57.613889 23691 raft_consensus.cc:348] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } }
I20260926 07:05:57.614059 23691 raft_consensus.cc:374] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:57.614112 23691 raft_consensus.cc:729] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e69c423c0dd94134b8ec3970b294dccb, State: Initialized, Role: FOLLOWER
I20260926 07:05:57.614231 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23623
I20260926 07:05:57.614260 23691 consensus_queue.cc:260] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } }
I20260926 07:05:57.614352 23691 raft_consensus.cc:388] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:57.614372 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:57.614391 23691 raft_consensus.cc:482] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:57.614445 23691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:57.614560 1893 external_mini_cluster.cc:1428] Setting key d571fddaabde650bd71fa002a117bcaf
I20260926 07:05:57.615657 23691 raft_consensus.cc:504] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } }
I20260926 07:05:57.615772 23691 leader_election.cc:302] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e69c423c0dd94134b8ec3970b294dccb; no voters:
I20260926 07:05:57.616253 23696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:57.616256 23691 leader_election.cc:288] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:57.616430 23696 raft_consensus.cc:686] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [term 1 LEADER]: Becoming Leader. State: Replica: e69c423c0dd94134b8ec3970b294dccb, State: Running, Role: LEADER
I20260926 07:05:57.616506 23696 consensus_queue.cc:237] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } }
I20260926 07:05:57.616567 23691 sys_catalog.cc:564] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:57.616839 23696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e69c423c0dd94134b8ec3970b294dccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } } }
I20260926 07:05:57.616910 23696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:57.617023 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:57.615078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49239 (local address 127.1.217.126:34781)
0926 07:05:57.615315 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:57.615329 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:57.615370 (+ 41us) server_negotiation.cc:407] Connection header received
0926 07:05:57.615488 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:57.615497 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:57.615527 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:57.615683 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:57.616344 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:57.616348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:57.616348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:57.616353 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:57.616409 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:57.616440 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:57.616508 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:57.616718 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 07:05:57.616756 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":104,"threads_started":1}
I20260926 07:05:57.617134 23700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:57.617177 23698 sys_catalog.cc:455] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e69c423c0dd94134b8ec3970b294dccb. Latest consensus state: current_term: 1 leader_uuid: "e69c423c0dd94134b8ec3970b294dccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e69c423c0dd94134b8ec3970b294dccb" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34781 } } }
I20260926 07:05:57.617240 23698 sys_catalog.cc:458] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:57.617661 23700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:05:57.618146 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:57.619228 23700 catalog_manager.cc:1347] Generated new cluster ID: 389761de3ecd41c09997a1701aa65745
I20260926 07:05:57.619554 23700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:57.620857 23695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:57.619659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43625 (local address 127.1.217.126:34781)
0926 07:05:57.619695 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:57.619697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:57.619923 (+ 226us) server_negotiation.cc:407] Connection header received
0926 07:05:57.619982 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:57.619983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:57.619988 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:57.620036 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:57.620641 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:57.620642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:57.620642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:05:57.620646 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:57.620659 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:05:57.620663 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:57.620716 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:57.620758 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 07:05:57.620789 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:05:57.660672 23700 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:57.660917 23700 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:57.664882 23700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e69c423c0dd94134b8ec3970b294dccb: Generated new TSK 0
I20260926 07:05:57.665091 23700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:57.670775 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23623
2026-09-26T07:05:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:57 dist-test-slave-t1mc krb5kdc[23722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:57 dist-test-slave-t1mc krb5kdc[23722](info): set up 2 sockets
Sep 26 07:05:57 dist-test-slave-t1mc krb5kdc[23722](info): commencing operation
krb5kdc: starting...
W20260926 07:05:58.708894 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:05:58 dist-test-slave-t1mc krb5kdc[23722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:05:58.823874 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34255
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:05:58.902866 23735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:05:58.903093 23735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:05:58.903131 23735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:05:58.904270 23735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:05:58.904335 23735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:05:58.904358 23735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:05:58.904388 23735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:05:58.904412 23735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:05:58.905915 23735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:34255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:05:58.906186 23735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:05:58.906400 23735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:05:58.911104 23735 server_base.cc:1034] running on GCE node
W20260926 07:05:58.911083 23743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:58.911089 23741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:05:58.911110 23740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:05:58.911576 23735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:05:58.911857 23735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:05:58.912999 23735 hybrid_clock.cc:648] HybridClock initialized: now 1790406358912980 us; error 35 us; skew 500 ppm
Sep 26 07:05:58 dist-test-slave-t1mc krb5kdc[23722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406358, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:05:58.915340 23735 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:05:58.915751 23735 webserver.cc:460] Webserver started at http://127.1.217.126:40635/ using document root <none> and password file <none>
I20260926 07:05:58.915962 23735 fs_manager.cc:359] Metadata directory not provided
I20260926 07:05:58.916006 23735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:05:58.916144 23735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:05:58.917232 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "ee3431da91194a309675831f97e51957"
format_stamp: "Formatted at 2026-09-26 07:05:58 on dist-test-slave-t1mc"
server_key: "b891f32da0b27aea5710c7dc7ef81b13"
server_key_iv: "445cb2bc06b08078914cb02516c085a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:58.917579 23735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "ee3431da91194a309675831f97e51957"
format_stamp: "Formatted at 2026-09-26 07:05:58 on dist-test-slave-t1mc"
server_key: "b891f32da0b27aea5710c7dc7ef81b13"
server_key_iv: "445cb2bc06b08078914cb02516c085a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:58.918910 23735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:05:58.919585 23750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:05:58.919771 23735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:05:58.919862 23735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "ee3431da91194a309675831f97e51957"
format_stamp: "Formatted at 2026-09-26 07:05:58 on dist-test-slave-t1mc"
server_key: "b891f32da0b27aea5710c7dc7ef81b13"
server_key_iv: "445cb2bc06b08078914cb02516c085a5"
server_key_version: "encryptionkey@0"
I20260926 07:05:58.919935 23735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:05:58.936626 23735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:05:58.938588 23735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:05:58.938728 23735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:05:58.942554 23735 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:34255
I20260926 07:05:58.942586 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:34255 every 8 connection(s)
I20260926 07:05:58.942930 23735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:05:58.943748 23803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:05:58.946004 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957: Bootstrap starting.
I20260926 07:05:58.946559 23803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957: Neither blocks nor log segments found. Creating new log.
I20260926 07:05:58.946866 23803 log.cc:824] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957: Log is configured to *not* fsync() on all Append() calls
I20260926 07:05:58.947569 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957: No bootstrap required, opened a new log
I20260926 07:05:58.948266 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23735
I20260926 07:05:58.948348 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:05:58.948536 1893 external_mini_cluster.cc:1428] Setting key 92bbd9078a9850c07d3aedf654d23139
I20260926 07:05:58.949827 23803 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } }
I20260926 07:05:58.950088 23803 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:05:58.951692 23803 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee3431da91194a309675831f97e51957, State: Initialized, Role: FOLLOWER
I20260926 07:05:58.951849 23803 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } }
I20260926 07:05:58.951939 23803 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:05:58.951977 23803 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:05:58.952014 23803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:05:58.952685 23803 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } }
I20260926 07:05:58.952808 23803 leader_election.cc:302] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee3431da91194a309675831f97e51957; no voters:
I20260926 07:05:58.952965 23803 leader_election.cc:288] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:05:58.953122 23808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:05:58.953225 23803 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:05:58.953274 23808 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [term 1 LEADER]: Becoming Leader. State: Replica: ee3431da91194a309675831f97e51957, State: Running, Role: LEADER
I20260926 07:05:58.953359 23808 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } }
I20260926 07:05:58.953692 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee3431da91194a309675831f97e51957. Latest consensus state: current_term: 1 leader_uuid: "ee3431da91194a309675831f97e51957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } } }
I20260926 07:05:58.953790 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:58.953950 23809 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee3431da91194a309675831f97e51957" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee3431da91194a309675831f97e51957" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 34255 } } }
I20260926 07:05:58.954010 23809 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957 [sys.catalog]: This master's current role is: LEADER
I20260926 07:05:58.954104 23817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:05:58.954587 23817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 07:05:58 dist-test-slave-t1mc krb5kdc[23722](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:05:58.956254 23817 catalog_manager.cc:1347] Generated new cluster ID: 245e323e5b864a099f76f3af7ce5b4ef
I20260926 07:05:58.956735 23817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:05:58.960249 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:58.949035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49777 (local address 127.1.217.126:34255)
0926 07:05:58.949227 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:58.949239 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:58.949275 (+ 36us) server_negotiation.cc:407] Connection header received
0926 07:05:58.949811 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:58.949820 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:58.949886 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:58.950032 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:58.950854 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:58.952355 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:58.953488 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:58.954141 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:58.954266 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:58.957064 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:58.957067 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:58.957067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:58.957079 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:58.958768 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:58.959130 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:58.959133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:58.959133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:58.959250 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:58.959476 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:58.959477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:58.959477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:58.959711 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:58.959891 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:58.959954 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 07:05:58.960092 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260926 07:05:58.961290 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:05:58.974965 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:05:58.962582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34007 (local address 127.1.217.126:34255)
0926 07:05:58.962622 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:05:58.962624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:05:58.968944 (+ 6320us) server_negotiation.cc:407] Connection header received
0926 07:05:58.969020 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:05:58.969021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:05:58.969043 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:05:58.969112 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:05:58.969649 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:58.970151 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:58.970886 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:05:58.971108 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:05:58.971197 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:05:58.972378 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:05:58.972379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:05:58.972379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:05:58.972388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:05:58.973830 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:58.974191 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:58.974192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:58.974193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:58.974252 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:05:58.974450 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:05:58.974451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:05:58.974452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:05:58.974564 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:05:58.974681 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 07:05:58.974779 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 07:05:58.974855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:05:58.982865 23817 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:05:58.983566 23817 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:05:58.994038 23817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee3431da91194a309675831f97e51957: Generated new TSK 0
I20260926 07:05:58.994237 23817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:05:59.001627 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23735
2026-09-26T07:05:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:05:59 dist-test-slave-t1mc krb5kdc[23834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:05:59 dist-test-slave-t1mc krb5kdc[23834](info): set up 2 sockets
Sep 26 07:05:59 dist-test-slave-t1mc krb5kdc[23834](info): commencing operation
krb5kdc: starting...
W20260926 07:06:00.029068 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:06:00 dist-test-slave-t1mc krb5kdc[23834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:06:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:06:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:06:00.147305 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43731
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:37489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:06:00.230573 23847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:00.230829 23847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:00.230867 23847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:00.232014 23847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:00.232085 23847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:00.232111 23847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:00.232141 23847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:00.232164 23847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:00.233664 23847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:37489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:00.233924 23847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:00.234148 23847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:00.238894 23855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:00.238894 23853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:00.238890 23852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:00.239061 23847 server_base.cc:1034] running on GCE node
I20260926 07:06:00.239387 23847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:00.239697 23847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:00.240844 23847 hybrid_clock.cc:648] HybridClock initialized: now 1790406360240820 us; error 37 us; skew 500 ppm
Sep 26 07:06:00 dist-test-slave-t1mc krb5kdc[23834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406360, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:00.243223 23847 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:06:00.243669 23847 webserver.cc:460] Webserver started at http://127.1.217.126:39521/ using document root <none> and password file <none>
I20260926 07:06:00.243865 23847 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:00.243906 23847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:00.243992 23847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:00.245090 23847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "24b0a92785a14d568a78e9c5ca491977"
format_stamp: "Formatted at 2026-09-26 07:06:00 on dist-test-slave-t1mc"
server_key: "fd0a509887e557457788ba0dbe26e3ff"
server_key_iv: "0b1e431516b62b1c6f36bd26edb1d6d7"
server_key_version: "encryptionkey@0"
I20260926 07:06:00.245393 23847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "24b0a92785a14d568a78e9c5ca491977"
format_stamp: "Formatted at 2026-09-26 07:06:00 on dist-test-slave-t1mc"
server_key: "fd0a509887e557457788ba0dbe26e3ff"
server_key_iv: "0b1e431516b62b1c6f36bd26edb1d6d7"
server_key_version: "encryptionkey@0"
I20260926 07:06:00.246652 23847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:06:00.247351 23862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:00.247547 23847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:06:00.247632 23847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "24b0a92785a14d568a78e9c5ca491977"
format_stamp: "Formatted at 2026-09-26 07:06:00 on dist-test-slave-t1mc"
server_key: "fd0a509887e557457788ba0dbe26e3ff"
server_key_iv: "0b1e431516b62b1c6f36bd26edb1d6d7"
server_key_version: "encryptionkey@0"
I20260926 07:06:00.247690 23847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:00.254516 23847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:00.256856 23847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:00.256971 23847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:00.260613 23847 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:43731
I20260926 07:06:00.260670 23914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:43731 every 8 connection(s)
I20260926 07:06:00.261065 23847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:06:00.261581 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23847
I20260926 07:06:00.261705 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:06:00.261901 1893 external_mini_cluster.cc:1428] Setting key d7207ab2adcf7d6f5da29027940cc9d5
I20260926 07:06:00.262135 23915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:00.265640 23915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977: Bootstrap starting.
I20260926 07:06:00.266346 23915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:00.266650 23915 log.cc:824] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:00.267465 23915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977: No bootstrap required, opened a new log
I20260926 07:06:00.268187 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:00.262666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47751 (local address 127.1.217.126:43731)
0926 07:06:00.262841 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:00.262850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:00.262884 (+ 34us) server_negotiation.cc:407] Connection header received
0926 07:06:00.263337 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:00.263345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:00.263406 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:00.263563 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:00.263750 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:00.265183 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:00.265980 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:00.266649 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:00.266763 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:00.267510 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:00.267514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:00.267514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:06:00.267719 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:00.267754 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:06:00.267768 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:00.267836 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:00.267897 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 07:06:00.267960 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 07:06:00.269310 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:06:00.269763 23915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } }
I20260926 07:06:00.269918 23915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:06:00.269963 23915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b0a92785a14d568a78e9c5ca491977, State: Initialized, Role: FOLLOWER
I20260926 07:06:00.270088 23915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } }
I20260926 07:06:00.270177 23915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:06:00.270219 23915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:06:00.270274 23915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:00.270980 23915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } }
I20260926 07:06:00.271090 23915 leader_election.cc:302] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b0a92785a14d568a78e9c5ca491977; no voters:
I20260926 07:06:00.271255 23915 leader_election.cc:288] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:06:00.271324 23925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:06:00.271495 23915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:00.271509 23925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [term 1 LEADER]: Becoming Leader. State: Replica: 24b0a92785a14d568a78e9c5ca491977, State: Running, Role: LEADER
I20260926 07:06:00.271636 23925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } }
I20260926 07:06:00.271944 23925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b0a92785a14d568a78e9c5ca491977. Latest consensus state: current_term: 1 leader_uuid: "24b0a92785a14d568a78e9c5ca491977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } } }
I20260926 07:06:00.272019 23925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:00.272163 23926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b0a92785a14d568a78e9c5ca491977" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b0a92785a14d568a78e9c5ca491977" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 43731 } } }
I20260926 07:06:00.272228 23926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977 [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:00.272351 23932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:06:00.273077 23932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:06:00.274744 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:00.270698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39633 (local address 127.1.217.126:43731)
0926 07:06:00.270734 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:00.270736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:00.270828 (+ 92us) server_negotiation.cc:407] Connection header received
0926 07:06:00.270882 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:00.270883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:00.270895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:00.270954 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:00.271039 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:00.271710 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:00.272935 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:00.273258 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:00.273327 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:00.274483 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:00.274484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:00.274485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:06:00.274491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:00.274505 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:06:00.274509 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:00.274574 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:00.274647 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 07:06:00.274670 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:06:00.275035 23932 catalog_manager.cc:1347] Generated new cluster ID: dda582be357149d284cec9a4506e822f
I20260926 07:06:00.275084 23932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:06:00.293020 23932 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:06:00.293776 23932 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:06:00.302452 23932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b0a92785a14d568a78e9c5ca491977: Generated new TSK 0
I20260926 07:06:00.302664 23932 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:06:00.306463 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23847
2026-09-26T07:06:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:06:00 dist-test-slave-t1mc krb5kdc[23946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:06:00 dist-test-slave-t1mc krb5kdc[23946](info): set up 2 sockets
Sep 26 07:06:00 dist-test-slave-t1mc krb5kdc[23946](info): commencing operation
krb5kdc: starting...
W20260926 07:06:01.332851 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:06:01 dist-test-slave-t1mc krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:06:01.451818 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46467
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:33569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:46467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:06:01.537287 23959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:01.537487 23959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:01.537510 23959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:01.538678 23959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:01.538724 23959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:01.538738 23959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:01.538777 23959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:01.538805 23959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:01.540254 23959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:33569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:46467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:46467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:01.540472 23959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:01.540681 23959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:01.545550 23965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:01.545539 23964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:01.545560 23959 server_base.cc:1034] running on GCE node
W20260926 07:06:01.545535 23967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:01.546032 23959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:01.546337 23959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:01.547506 23959 hybrid_clock.cc:648] HybridClock initialized: now 1790406361547496 us; error 32 us; skew 500 ppm
Sep 26 07:06:01 dist-test-slave-t1mc krb5kdc[23946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406361, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:01.555415 23959 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:06:01.555825 23959 webserver.cc:460] Webserver started at http://127.1.217.126:37345/ using document root <none> and password file <none>
I20260926 07:06:01.556018 23959 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:01.556061 23959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:01.556192 23959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:01.557338 23959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "de52f53f092e4360b590fd07b0b06bce"
format_stamp: "Formatted at 2026-09-26 07:06:01 on dist-test-slave-t1mc"
server_key: "5b52ba0c21ddf734d96a3b845ed96767"
server_key_iv: "1fb166c3fa232c00239c87f8f4a4c292"
server_key_version: "encryptionkey@0"
I20260926 07:06:01.557663 23959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "de52f53f092e4360b590fd07b0b06bce"
format_stamp: "Formatted at 2026-09-26 07:06:01 on dist-test-slave-t1mc"
server_key: "5b52ba0c21ddf734d96a3b845ed96767"
server_key_iv: "1fb166c3fa232c00239c87f8f4a4c292"
server_key_version: "encryptionkey@0"
I20260926 07:06:01.559012 23959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 07:06:01.559808 23974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:01.560025 23959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:06:01.560119 23959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "de52f53f092e4360b590fd07b0b06bce"
format_stamp: "Formatted at 2026-09-26 07:06:01 on dist-test-slave-t1mc"
server_key: "5b52ba0c21ddf734d96a3b845ed96767"
server_key_iv: "1fb166c3fa232c00239c87f8f4a4c292"
server_key_version: "encryptionkey@0"
I20260926 07:06:01.560184 23959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:01.563733 23959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:01.565805 23959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:01.565914 23959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:01.569943 23959 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:46467
I20260926 07:06:01.569993 24026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:46467 every 8 connection(s)
I20260926 07:06:01.570317 23959 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:06:01.571343 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:01.573650 24027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: Bootstrap starting.
I20260926 07:06:01.574222 24027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:01.574512 24027 log.cc:824] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:01.575188 24027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: No bootstrap required, opened a new log
I20260926 07:06:01.576546 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 23959
I20260926 07:06:01.576689 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:06:01.576930 1893 external_mini_cluster.cc:1428] Setting key 717890260bf7dd1ef34011ae74f34d4d
I20260926 07:06:01.576897 24027 raft_consensus.cc:348] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } }
I20260926 07:06:01.577095 24027 raft_consensus.cc:374] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 07:06:01.577138 24027 raft_consensus.cc:729] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de52f53f092e4360b590fd07b0b06bce, State: Initialized, Role: FOLLOWER
I20260926 07:06:01.577277 24027 consensus_queue.cc:260] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } }
I20260926 07:06:01.577363 24027 raft_consensus.cc:388] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 07:06:01.577395 24027 raft_consensus.cc:482] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 07:06:01.577442 24027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:01.578070 24027 raft_consensus.cc:504] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } }
I20260926 07:06:01.578203 24027 leader_election.cc:302] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de52f53f092e4360b590fd07b0b06bce; no voters:
I20260926 07:06:01.578370 24027 leader_election.cc:288] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 07:06:01.578423 24032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:06:01.578541 24032 raft_consensus.cc:686] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [term 1 LEADER]: Becoming Leader. State: Replica: de52f53f092e4360b590fd07b0b06bce, State: Running, Role: LEADER
I20260926 07:06:01.578665 24027 sys_catalog.cc:564] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:01.578672 24032 consensus_queue.cc:237] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } }
I20260926 07:06:01.579334 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:01.577407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45099 (local address 127.1.217.126:46467)
0926 07:06:01.577600 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:01.577611 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:01.577648 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:06:01.577992 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:01.577997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:01.578020 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:01.578134 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:01.578836 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:01.578840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:01.578840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:06:01.578844 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:01.578884 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:06:01.578896 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:01.579065 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:01.579099 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:06:01.579129 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 07:06:01.579384 24033 sys_catalog.cc:455] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [sys.catalog]: SysCatalogTable state changed. Reason: New leader de52f53f092e4360b590fd07b0b06bce. Latest consensus state: current_term: 1 leader_uuid: "de52f53f092e4360b590fd07b0b06bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } } }
I20260926 07:06:01.579478 24033 sys_catalog.cc:458] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:01.579290 24034 sys_catalog.cc:455] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de52f53f092e4360b590fd07b0b06bce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de52f53f092e4360b590fd07b0b06bce" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 46467 } } }
I20260926 07:06:01.579716 24034 sys_catalog.cc:458] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:01.580318 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 07:06:01.580696 24048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:01.580737 24048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:06:01.580844 24041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:06:01.581012 24041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:06:01.582599 24041 catalog_manager.cc:1347] Generated new cluster ID: e3b4bb16bd314548b3b2e30904a0c3ae
I20260926 07:06:01.582651 24041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:06:01.582863 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:01.581785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57797 (local address 127.1.217.126:46467)
0926 07:06:01.581824 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:01.581826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:01.581968 (+ 142us) server_negotiation.cc:407] Connection header received
0926 07:06:01.582040 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:01.582040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:01.582045 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:01.582099 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:01.582656 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:01.582657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:01.582658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 07:06:01.582662 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:01.582677 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 07:06:01.582682 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:01.582734 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:01.582793 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 07:06:01.582806 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:06:01.597118 24041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:06:01.597299 24041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:06:01.608131 24041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de52f53f092e4360b590fd07b0b06bce: Generated new TSK 0
I20260926 07:06:01.608338 24041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:06:01.613451 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 23959
2026-09-26T07:06:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1306 ms)
[----------] 18 tests from AuthTokenIssuingTest (23643 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:06:01 dist-test-slave-t1mc krb5kdc[24058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:06:01 dist-test-slave-t1mc krb5kdc[24058](info): set up 2 sockets
Sep 26 07:06:01 dist-test-slave-t1mc krb5kdc[24058](info): commencing operation
krb5kdc: starting...
W20260926 07:06:02.661041 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:06:02 dist-test-slave-t1mc krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:06:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:06:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:06:02.779165 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42549
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:06:02.859118 24071 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:02.859375 24071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:02.859405 24071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:02.859447 24071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:02.860579 24071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:02.860649 24071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:02.860671 24071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:02.860692 24071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:02.860720 24071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:02.862249 24071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:02.862536 24071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:02.862762 24071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:02.867726 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:02.867870 24071 server_base.cc:1034] running on GCE node
W20260926 07:06:02.867726 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:02.867981 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:02.868251 24071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:02.868569 24071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:02.869737 24071 hybrid_clock.cc:648] HybridClock initialized: now 1790406362869712 us; error 80 us; skew 500 ppm
Sep 26 07:06:02 dist-test-slave-t1mc krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:02.878738 24071 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:06:02.879230 24071 webserver.cc:460] Webserver started at http://127.1.217.126:42769/ using document root <none> and password file <none>
I20260926 07:06:02.879422 24071 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:02.879462 24071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:02.879554 24071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:02.880582 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "e8e38666515e46e9a1eb5fc4811c215f"
format_stamp: "Formatted at 2026-09-26 07:06:02 on dist-test-slave-t1mc"
server_key: "d3e59c11afc3a1eccb7a223ab18ec512"
server_key_iv: "bd28e2089a1ea5e044132cb0fd28ad75"
server_key_version: "encryptionkey@0"
I20260926 07:06:02.880996 24071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "e8e38666515e46e9a1eb5fc4811c215f"
format_stamp: "Formatted at 2026-09-26 07:06:02 on dist-test-slave-t1mc"
server_key: "d3e59c11afc3a1eccb7a223ab18ec512"
server_key_iv: "bd28e2089a1ea5e044132cb0fd28ad75"
server_key_version: "encryptionkey@0"
I20260926 07:06:02.882388 24071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:06:02.883167 24086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:02.883404 24071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:06:02.883513 24071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "e8e38666515e46e9a1eb5fc4811c215f"
format_stamp: "Formatted at 2026-09-26 07:06:02 on dist-test-slave-t1mc"
server_key: "d3e59c11afc3a1eccb7a223ab18ec512"
server_key_iv: "bd28e2089a1ea5e044132cb0fd28ad75"
server_key_version: "encryptionkey@0"
I20260926 07:06:02.883600 24071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:02.894632 24071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:02.896556 24071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:02.896710 24071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:02.900449 24071 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:42549
I20260926 07:06:02.900518 24138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:42549 every 8 connection(s)
I20260926 07:06:02.900918 24071 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:06:02.901870 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:02.903317 24139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:02.903671 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24071
I20260926 07:06:02.903782 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:06:02.904000 1893 external_mini_cluster.cc:1428] Setting key f9cfb63b85e98bc6e15008109ba4ef38
Sep 26 07:06:02 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.217.125@KRBTEST.COM; defaulting to no policy
I20260926 07:06:02.914809 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.904006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42549 (local address 127.0.0.1:36792)
0926 07:06:02.904191 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:02.904237 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:02.904984 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:02.905281 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:02.905284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:02.905650 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:02.905804 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:02.905809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:02.907182 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:02.907183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:02.907913 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:02.907915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:02.908677 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:02.908678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:02.908733 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:02.909348 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:02.909358 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:02.911052 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:02.913081 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:02.913088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:02.913089 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:02.913339 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:02.913633 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:02.913634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:02.913634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:02.913729 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:02.914220 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:02.914223 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:02.914276 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 07:06:02.914366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 07:06:02.914531 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260926 07:06:02.914813 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.904055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.1.217.126:42549)
0926 07:06:02.904198 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:02.904207 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:02.904313 (+ 106us) server_negotiation.cc:407] Connection header received
0926 07:06:02.905142 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:02.905144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:02.905161 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:02.905228 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:02.905910 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:02.907069 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:02.908000 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:02.908557 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:02.908681 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:02.911159 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:02.911165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:02.911166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:02.911172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:02.913010 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:02.913444 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:02.913447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:02.913448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:02.913554 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:02.913798 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:02.913799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:02.913799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:02.914089 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:02.914193 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:02.914414 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 07:06:02.914658 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.125@KRBTEST.COM" created.
I20260926 07:06:02.915890 24139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:02.916296 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.916167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48020)
0926 07:06:02.916224 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:02.916267 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 07:06:02.916419 24089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
I20260926 07:06:02.916838 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.916705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48036)
0926 07:06:02.916760 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:02.916808 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:06:02.916929 24139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:41909: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
I20260926 07:06:02.933995 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:41909
--webserver_interface=127.1.217.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:06:02.966142 24139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } attempt: 1
I20260926 07:06:02.966647 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.966505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48046)
0926 07:06:02.966569 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:02.966608 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 07:06:02.966981 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:02.966887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48054)
0926 07:06:02.966914 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:02.966958 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 07:06:02.967058 24139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:41909: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
W20260926 07:06:03.023867 24146 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:03.024140 24146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:03.024173 24146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:03.024219 24146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:03.025429 24146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:03.025506 24146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:03.025532 24146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:03.025552 24146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:03.025580 24146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:03.027103 24146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:41909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.217.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:03.027391 24146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:03.027618 24146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:03.032313 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:03.032312 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:03.032568 24146 server_base.cc:1034] running on GCE node
W20260926 07:06:03.032725 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:03.033031 24146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:03.033290 24146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:03.034437 24146 hybrid_clock.cc:648] HybridClock initialized: now 1790406363034426 us; error 34 us; skew 500 ppm
I20260926 07:06:03.035197 24139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } attempt: 2
I20260926 07:06:03.035631 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.035492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48062)
0926 07:06:03.035558 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.035594 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:06:03.036005 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.035893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48066)
0926 07:06:03.035943 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.035968 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 07:06:03.036140 24139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:41909: Network error: Client connection negotiation failed: client connection to 127.1.217.125:41909: connect: Connection refused (error 111)
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:03.037298 24146 init.cc:377] Logged in from keytab as kudu/127.1.217.125@KRBTEST.COM (short username kudu)
I20260926 07:06:03.037721 24146 webserver.cc:460] Webserver started at http://127.1.217.125:35963/ using document root <none> and password file <none>
I20260926 07:06:03.037966 24146 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:03.038038 24146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:03.038189 24146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:03.039266 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data/instance:
uuid: "03b9daf87f8747dd8a1616354ecd11a7"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "c3fcf48bf3e167047d987c4d749e3cc4"
server_key_iv: "a785dd93a5db24cd0437c8f4076f9e44"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.039626 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal/instance:
uuid: "03b9daf87f8747dd8a1616354ecd11a7"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "c3fcf48bf3e167047d987c4d749e3cc4"
server_key_iv: "a785dd93a5db24cd0437c8f4076f9e44"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.040988 24146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:06:03.041725 24161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:03.041904 24146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:06:03.042018 24146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "03b9daf87f8747dd8a1616354ecd11a7"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "c3fcf48bf3e167047d987c4d749e3cc4"
server_key_iv: "a785dd93a5db24cd0437c8f4076f9e44"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.042099 24146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:03.054654 24146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:03.057066 24146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:03.057226 24146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:03.061192 24146 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.125:41909
I20260926 07:06:03.061242 24213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.125:41909 every 8 connection(s)
I20260926 07:06:03.061589 24146 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
I20260926 07:06:03.062573 24214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:03.064107 24214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:03.069478 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24146
I20260926 07:06:03.069587 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-1/wal/instance
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:03.069808 1893 external_mini_cluster.cc:1428] Setting key e9d6dea1d9cb4d2e57b256675eb416ee
I20260926 07:06:03.073539 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.064856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.1.217.126:42549)
0926 07:06:03.064925 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.064927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.065145 (+ 218us) server_negotiation.cc:407] Connection header received
0926 07:06:03.065644 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.065645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.065655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.065710 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.066255 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.066735 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.067664 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.067859 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.067911 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.070288 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.070290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.070290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.070297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.072038 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.072453 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.072455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.072455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.072521 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.072802 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.072804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.072804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.072968 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.073040 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.073299 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.073398 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:06:03.073589 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.064821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42549 (local address 127.0.0.1:36804)
0926 07:06:03.065023 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.065059 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.065562 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.065761 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.065764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.065997 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.066185 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.066189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.066815 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.066816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.067591 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.067593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.067939 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.067940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.068040 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.068496 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.068504 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.070191 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.072139 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.072146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.072148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.072374 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.072594 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.072595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.072595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.072692 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.073036 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.073041 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.073161 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.073248 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.073373 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260926 07:06:03.074191 24214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.1.217.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.124@KRBTEST.COM" created.
I20260926 07:06:03.081825 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.074466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48072)
0926 07:06:03.074523 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.074536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.074576 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.074889 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.074890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.075036 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.075085 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.075087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.075962 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.075963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.076462 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.076463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.077015 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.077016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.077078 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.077516 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.077521 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.078805 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.080626 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.080627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.080627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.080867 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.081086 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.081087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.081087 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.081143 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.081452 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.081453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.081493 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.081566 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.081669 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:06:03.081898 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.074547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.1.217.125:41909)
0926 07:06:03.074697 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.074719 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.074742 (+ 23us) server_negotiation.cc:407] Connection header received
0926 07:06:03.074763 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.074765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.074779 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.074872 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.075175 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.075861 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.076551 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.076924 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.077018 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.078922 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.078930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.078930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.078938 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.080553 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.080946 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.080948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.080949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.081006 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.081219 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.081220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.081221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.081377 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.081473 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.081597 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.081740 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260926 07:06:03.082593 24214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:03.082998 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.082852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41496)
0926 07:06:03.082910 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.082962 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 07:06:03.083159 24162 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
I20260926 07:06:03.083567 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.083484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41506)
0926 07:06:03.083518 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.083543 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 07:06:03.083707 24214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41325: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
I20260926 07:06:03.098017 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:41325
--webserver_interface=127.1.217.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:41403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:06:03.127298 24139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } attempt: 3
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:03.132941 24214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } attempt: 1
I20260926 07:06:03.133323 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.133201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41518)
0926 07:06:03.133260 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.133295 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 07:06:03.133724 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.133610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41520)
0926 07:06:03.133666 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.133695 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 07:06:03.133822 24214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41325: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
I20260926 07:06:03.134575 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.127765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.1.217.125:41909)
0926 07:06:03.127845 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.127847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.127859 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:06:03.127912 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.127913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.127928 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.128011 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.128354 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.128907 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.129661 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.129886 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.129962 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.131852 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.131853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.131853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.131861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.133426 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.133834 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.133836 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.133836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.133895 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.134099 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.134100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.134100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.134205 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.134259 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.134453 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.134523 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 07:06:03.134701 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.127698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48088)
0926 07:06:03.127774 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.127789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.127844 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.128019 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.128020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.128190 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.128258 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.128260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.128990 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.128991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.129546 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.129548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.129971 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.129971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.130056 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.130533 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.130537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.131758 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.133505 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.133506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.133507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.133742 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.133955 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.133956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.133956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.134030 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.134277 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.134278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.134342 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.134430 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.134556 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 07:06:03.135130 24139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:03.135495 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.135363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41536)
0926 07:06:03.135418 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.135459 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:06:03.135876 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.135774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41546)
0926 07:06:03.135832 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.135856 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 07:06:03.136008 24139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41325: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
I20260926 07:06:03.167163 24139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } attempt: 1
I20260926 07:06:03.167682 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.167546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41548)
0926 07:06:03.167612 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.167654 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:06:03.168089 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.168003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41562)
0926 07:06:03.168047 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.168073 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 07:06:03.168210 24139 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41325: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
W20260926 07:06:03.180384 24221 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:03.180596 24221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:03.180613 24221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:03.180634 24221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:03.181754 24221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:03.181797 24221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:03.181811 24221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:03.181824 24221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:03.181836 24221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:03.183372 24221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:41403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:42549,127.1.217.125:41909,127.1.217.124:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.217.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:03.183555 24221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:03.183717 24221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 07:06:03.188647 24221 server_base.cc:1034] running on GCE node
W20260926 07:06:03.188602 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:03.188639 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:03.188616 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:03.188997 24221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:03.189280 24221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:03.190452 24221 hybrid_clock.cc:648] HybridClock initialized: now 1790406363190433 us; error 33 us; skew 500 ppm
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:03.192852 24221 init.cc:377] Logged in from keytab as kudu/127.1.217.124@KRBTEST.COM (short username kudu)
I20260926 07:06:03.193321 24221 webserver.cc:460] Webserver started at http://127.1.217.124:41837/ using document root <none> and password file <none>
I20260926 07:06:03.193539 24221 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:03.193607 24221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:03.193758 24221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:03.194854 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data/instance:
uuid: "91907a1f5f6e4a1caea3563975f8cb66"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "61c2d32060958fc7c0d9d585c683e374"
server_key_iv: "a3403d0f5e1fc0b2b4f34a7ada4425dc"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.195209 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal/instance:
uuid: "91907a1f5f6e4a1caea3563975f8cb66"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "61c2d32060958fc7c0d9d585c683e374"
server_key_iv: "a3403d0f5e1fc0b2b4f34a7ada4425dc"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.196514 24221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 07:06:03.197394 24236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:03.197572 24221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 07:06:03.197672 24221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "91907a1f5f6e4a1caea3563975f8cb66"
format_stamp: "Formatted at 2026-09-26 07:06:03 on dist-test-slave-t1mc"
server_key: "61c2d32060958fc7c0d9d585c683e374"
server_key_iv: "a3403d0f5e1fc0b2b4f34a7ada4425dc"
server_key_version: "encryptionkey@0"
I20260926 07:06:03.197758 24221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:03.201953 24214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } attempt: 2
I20260926 07:06:03.202342 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.202220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41564)
0926 07:06:03.202276 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.202313 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 07:06:03.202677 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.202584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41568)
0926 07:06:03.202628 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.202651 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 07:06:03.202760 24214 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41325: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41325: connect: Connection refused (error 111)
I20260926 07:06:03.228941 24221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:03.230908 24221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:03.231092 24221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:03.235203 24221 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.124:41325
I20260926 07:06:03.235236 24288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.124:41325 every 8 connection(s)
I20260926 07:06:03.235577 24221 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
I20260926 07:06:03.236575 24289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:03.238153 24289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:03.243294 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24221
I20260926 07:06:03.243400 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790406101908495-1893-0/minicluster-data/master-2/wal/instance
I20260926 07:06:03.243404 24139 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } attempt: 2
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:03.243620 1893 external_mini_cluster.cc:1428] Setting key 4be8f90a4abfa5edeaf3ffafeca9c95e
I20260926 07:06:03.247289 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.238832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36820 (local address 127.1.217.126:42549)
0926 07:06:03.238906 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.238908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.239045 (+ 137us) server_negotiation.cc:407] Connection header received
0926 07:06:03.239493 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.239495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.239506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.239569 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.240168 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.240636 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.241619 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.241813 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.241870 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.244210 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.244212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.244212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.244219 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.245846 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.246278 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.246279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.246280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.246343 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.246607 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.246607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.246608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.246722 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.246793 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.247105 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.247193 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 07:06:03.247488 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.238754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:42549 (local address 127.0.0.1:36820)
0926 07:06:03.238915 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.238955 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.239386 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.239641 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.239644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.239879 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.240079 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.240083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.240713 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.240714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.241540 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.241542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.241901 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.241902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.242009 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.242451 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.242459 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.244027 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.245936 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.245949 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.245951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.246203 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.246417 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.246418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.246418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.246529 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.246791 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.246798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.246951 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.247058 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.247200 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260926 07:06:03.248050 24289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:03.252939 24141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.243751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41572)
0926 07:06:03.243809 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.243819 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.243854 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.244156 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.244158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.244381 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.244424 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.244426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.245922 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.245923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.247113 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.247115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.247718 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.247719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.247773 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.248146 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.248151 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.249628 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.251703 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.251703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.251704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.251930 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.252189 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.252190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.252190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.252280 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.252591 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.252592 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.252638 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.252715 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.252827 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:06:03.253264 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.243814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.1.217.124:41325)
0926 07:06:03.244014 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.244019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.244037 (+ 18us) server_negotiation.cc:407] Connection header received
0926 07:06:03.244057 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.244059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.244072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.244146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.244513 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.245787 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.247200 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.247632 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.247906 (+ 274us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.249709 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.249715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.249716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.249722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.251634 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.252036 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.252039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.252039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.252106 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.252336 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.252337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.252337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.252507 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.252596 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.252785 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.252925 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260926 07:06:03.254129 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.243973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41574 (local address 127.1.217.124:41325)
0926 07:06:03.244093 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.244094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.244198 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:06:03.244255 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.244256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.244266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.244346 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.245134 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.245852 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.246902 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.247345 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.247413 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.250440 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.250441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.250441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.250448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.252669 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.253117 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.253118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.253118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.253165 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.253360 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.253361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.253361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.253473 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.253581 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.253977 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.254055 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260926 07:06:03.255285 24139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: Bootstrap starting.
I20260926 07:06:03.256021 24139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:03.256127 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.248315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:41909 (local address 127.0.0.1:48104)
0926 07:06:03.248794 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.248805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.248859 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.249016 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.249017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.249165 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.249223 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.249224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.250156 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.250157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.250933 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.250936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.251332 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.251333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.251400 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.251760 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.251765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.253136 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.255125 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.255126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.255126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.255346 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.255534 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.255535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.255535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.255589 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.255849 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.255849 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.255888 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.255956 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.256042 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432}
I20260926 07:06:03.256147 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.248363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48104 (local address 127.1.217.125:41909)
0926 07:06:03.248403 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.248404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.248858 (+ 454us) server_negotiation.cc:407] Connection header received
0926 07:06:03.248932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.248933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.248944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.248997 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.249308 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.250063 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.251043 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.251251 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.251323 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.253228 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.253229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.253230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.253237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.255038 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.255404 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.255405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.255406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.255460 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.255661 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.255662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.255662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.255774 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.255858 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.255986 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.256057 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:06:03.256356 24139 log.cc:824] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:03.256610 24289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:03.257391 24139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: No bootstrap required, opened a new log
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:03.260126 24139 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.260238 24139 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e38666515e46e9a1eb5fc4811c215f, State: Initialized, Role: FOLLOWER
I20260926 07:06:03.260320 24139 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:03.260617 24296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.260689 24139 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:03.260703 24296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:03.263648 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.256879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41590)
0926 07:06:03.256942 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.256951 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.256985 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.257190 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.257190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.257326 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.257349 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.257350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.257932 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.257933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.258384 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.258385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.258759 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.258759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.258813 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.259199 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.259203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.260699 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.262656 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.262657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.262658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.262864 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.263034 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.263035 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.263035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.263093 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.263372 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.263373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.263409 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.263453 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.263532 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 07:06:03.263953 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.257046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.1.217.124:41325)
0926 07:06:03.257078 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.257079 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.257085 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:06:03.257101 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.257101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.257116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.257187 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.257421 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.257848 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.258474 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.258680 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.258759 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.260838 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.260839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.260839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.260847 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.262574 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.262931 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.262932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.262933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.262985 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.263158 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.263159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.263159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.263302 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.263760 (+ 458us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.263804 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.263882 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:06:03.264139 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.254701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36836 (local address 127.1.217.126:42549)
0926 07:06:03.254736 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.254738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.254747 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:06:03.254771 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.254772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.254782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.254837 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.255357 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.256013 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.257576 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.257827 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.257870 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.260917 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.260918 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.260919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.260926 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.262631 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.263057 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.263059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.263059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.263119 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.263312 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.263313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.263313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.263442 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.263592 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.263961 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.264056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 07:06:03.265718 24289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: Bootstrap starting.
I20260926 07:06:03.266512 24289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:03.266784 24289 log.cc:824] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:03.267607 24289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: No bootstrap required, opened a new log
I20260926 07:06:03.264884 24071 master_runner.cc:186] Error getting master registration for 127.1.217.125:41909: OK, instance_id { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" instance_seqno: 1790406363060000 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:06:03.267843 24071 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 07:06:03.268285 24307 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:03.268386 24307 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:06:03.269637 24289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.269775 24289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91907a1f5f6e4a1caea3563975f8cb66, State: Initialized, Role: FOLLOWER
I20260926 07:06:03.269862 24289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.270097 24310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.270159 24289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:03.270231 24310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:06:03.272230 24321 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:03.272301 24321 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:03.274937 24221 master_runner.cc:186] Error getting master registration for 127.1.217.125:41909: OK, instance_id { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" instance_seqno: 1790406363060000 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:06:03.277235 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.264978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.1.217.125:41909)
0926 07:06:03.267915 (+ 2937us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.267917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.267926 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:06:03.267950 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.267951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.267963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.268032 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.268639 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.269397 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.270382 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.270685 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.270757 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.273778 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.273779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.273779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.273787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.275693 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.276087 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.276088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.276088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.276142 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.276325 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.276326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.276326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.276449 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.276594 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.277061 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.277137 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2913}
I20260926 07:06:03.277237 24221 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:06:03.278779 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:06:03.279760 24113 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e8e38666515e46e9a1eb5fc4811c215f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36836
I20260926 07:06:03.279908 24113 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:06:03.279954 24113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:03.281001 24113 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.281229 24113 leader_election.cc:288] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [CANDIDATE]: Term 1 election: Requested vote from peers 03b9daf87f8747dd8a1616354ecd11a7 (127.1.217.125:41909), 91907a1f5f6e4a1caea3563975f8cb66 (127.1.217.124:41325)
I20260926 07:06:03.281554 24263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8e38666515e46e9a1eb5fc4811c215f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "91907a1f5f6e4a1caea3563975f8cb66"
I20260926 07:06:03.281661 24263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:03.282498 24263 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e38666515e46e9a1eb5fc4811c215f in term 1.
I20260926 07:06:03.283039 24087 leader_election.cc:302] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91907a1f5f6e4a1caea3563975f8cb66, e8e38666515e46e9a1eb5fc4811c215f; no voters:
I20260926 07:06:03.283201 24296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:06:03.283382 24296 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [term 1 LEADER]: Becoming Leader. State: Replica: e8e38666515e46e9a1eb5fc4811c215f, State: Running, Role: LEADER
I20260926 07:06:03.283471 24296 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.283879 24296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e38666515e46e9a1eb5fc4811c215f. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.283946 24296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:03.284572 24329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:06:03.284796 24329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:06:03.293884 24214 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } attempt: 3
I20260926 07:06:03.295245 24263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [term 1 FOLLOWER]: Refusing update from remote peer e8e38666515e46e9a1eb5fc4811c215f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:06:03.296175 24323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [LEADER]: Connected to new peer: Peer: permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:06:03.297699 24310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e38666515e46e9a1eb5fc4811c215f. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.297808 24310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 07:06:03 dist-test-slave-t1mc krb5kdc[24058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406363, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:03.298853 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.298938 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:03.299316 24310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.299396 24310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:03.299718 24332 mvcc.cc:205] Tried to move back new op lower bound from 7333504464049299456 to 7333504464009625600. Current Snapshot: MvccSnapshot[applied={T|T < 7333504464049299456}]
I20260926 07:06:03.300312 24334 mvcc.cc:205] Tried to move back new op lower bound from 7333504464049299456 to 7333504464009625600. Current Snapshot: MvccSnapshot[applied={T|T < 7333504464049299456}]
I20260926 07:06:03.302455 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.294148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41325 (local address 127.0.0.1:41608)
0926 07:06:03.294232 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:03.294242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:03.294282 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:03.294557 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:03.294558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:03.294713 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:03.294754 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.294756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.295701 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.295702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.296481 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:03.296483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.297018 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.297019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:03.297072 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.297651 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:03.297656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:03.299136 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:03.301245 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.301246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.301247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.301497 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.301696 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:03.301697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:03.301697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:03.301761 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:03.302076 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:03.302077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:03.302149 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 07:06:03.302226 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 07:06:03.302345 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 07:06:03.302518 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.294373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41608 (local address 127.1.217.124:41325)
0926 07:06:03.294414 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.294416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.294422 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:06:03.294443 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.294444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.294453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.294498 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.294841 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.295618 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.296584 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.296925 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.296976 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.299222 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.299223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.299223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.299230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.301146 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.301570 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.301571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.301572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.301626 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.301850 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.301851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.301851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.301986 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.302066 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.302350 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.302425 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":22}
I20260926 07:06:03.304481 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: Bootstrap starting.
I20260926 07:06:03.305258 24214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:03.305598 24214 log.cc:824] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:03.305919 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.285080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.1.217.125:41909)
0926 07:06:03.285113 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.285115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.292972 (+ 7857us) server_negotiation.cc:407] Connection header received
0926 07:06:03.293075 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.293077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.293087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.293135 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.294281 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.294957 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.299223 (+ 4266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.299541 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.299581 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.301524 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.301526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.301526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.301533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.303594 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.304281 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.304282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.304283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.304339 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.304738 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.304739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.304739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.304932 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.305028 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.305502 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.305576 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:06:03.307955 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.285535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41600 (local address 127.1.217.124:41325)
0926 07:06:03.285599 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.285601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.293707 (+ 8106us) server_negotiation.cc:407] Connection header received
0926 07:06:03.293799 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.293800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.293810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.293839 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.294909 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.295613 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.296600 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.296916 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.296960 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.304883 (+ 7923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.304884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.304885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.304891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.306590 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.306966 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.306967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.306967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.307021 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.307224 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.307225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.307226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.307360 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.307443 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.307760 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.307838 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260926 07:06:03.308305 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:03.284559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.1.217.126:42549)
0926 07:06:03.284596 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.284598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.292923 (+ 8325us) server_negotiation.cc:407] Connection header received
0926 07:06:03.293031 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.293032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.293045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.293078 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.293945 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.294693 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.295633 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:03.295904 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:03.295954 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:03.302788 (+ 6834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:03.302790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:03.302790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:03.302798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:03.304505 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.306991 (+ 2486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.306992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.306993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.307047 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:03.307375 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:03.307376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:03.307376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:03.307504 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:03.307576 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:03.308129 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 07:06:03.308207 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 07:06:03.308327 24329 catalog_manager.cc:1347] Generated new cluster ID: 908501e54efa49cb8264991193a47104
I20260926 07:06:03.308363 24329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:06:03.309868 24214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: No bootstrap required, opened a new log
I20260926 07:06:03.311913 24214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.312089 24214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b9daf87f8747dd8a1616354ecd11a7, State: Initialized, Role: FOLLOWER
I20260926 07:06:03.312198 24214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } }
I20260926 07:06:03.312551 24214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:03.312606 24188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e8e38666515e46e9a1eb5fc4811c215f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03b9daf87f8747dd8a1616354ecd11a7"
I20260926 07:06:03.312711 24188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:03.313517 24188 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8e38666515e46e9a1eb5fc4811c215f in term 1.
I20260926 07:06:03.313689 24187 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [term 1 FOLLOWER]: Refusing update from remote peer e8e38666515e46e9a1eb5fc4811c215f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:06:03.315186 24296 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [LEADER]: Connected to new peer: Peer: permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:06:03.312532 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.316867 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:03.316926 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e38666515e46e9a1eb5fc4811c215f. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.316967 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:03.316993 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.317028 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 07:06:03.317632 24350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:03.317682 24350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:06:03.319144 24352 mvcc.cc:205] Tried to move back new op lower bound from 7333504464049299456 to 7333504464009625600. Current Snapshot: MvccSnapshot[applied={T|T < 7333504464049299456}]
I20260926 07:06:03.327520 24296 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e8e38666515e46e9a1eb5fc4811c215f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e38666515e46e9a1eb5fc4811c215f" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 42549 } } peers { permanent_uuid: "03b9daf87f8747dd8a1616354ecd11a7" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 41909 } } peers { permanent_uuid: "91907a1f5f6e4a1caea3563975f8cb66" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41325 } } }
I20260926 07:06:03.327675 24296 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:03.335668 24329 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:06:03.336521 24329 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:06:03.349771 24329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8e38666515e46e9a1eb5fc4811c215f: Generated new TSK 0
I20260926 07:06:03.350077 24329 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 07:06:03.377684 24142 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.376029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.1.217.126:42549)
0926 07:06:03.376068 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.376069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.376856 (+ 787us) server_negotiation.cc:407] Connection header received
0926 07:06:03.377082 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.377083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.377094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.377124 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.377605 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36840: BlockingRecv error: recv got EOF from 127.0.0.1:36840 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 07:06:03.378439 24294 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.375773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.1.217.124:41325)
0926 07:06:03.375820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.375823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.377902 (+ 2079us) server_negotiation.cc:407] Connection header received
0926 07:06:03.377998 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.377999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.378009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.378038 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.378383 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41616: BlockingRecv error: recv got EOF from 127.0.0.1:41616 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 07:06:03.379153 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.376423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.1.217.125:41909)
0926 07:06:03.376471 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.376473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.378673 (+ 2200us) server_negotiation.cc:407] Connection header received
0926 07:06:03.378766 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.378768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.378779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.378807 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.379093 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48144: BlockingRecv error: recv got EOF from 127.0.0.1:48144 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:06:03.381513 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.380729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.1.217.125:41909)
0926 07:06:03.380763 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.380792 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.381004 (+ 212us) server_negotiation.cc:407] Connection header received
0926 07:06:03.381108 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.381109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.381127 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.381155 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.381467 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48146: BlockingRecv error: recv got EOF from 127.0.0.1:48146 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:06:03.384516 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.383784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.1.217.125:41909)
0926 07:06:03.383817 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.383819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.384041 (+ 222us) server_negotiation.cc:407] Connection header received
0926 07:06:03.384131 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.384132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.384141 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.384168 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.384472 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48156: BlockingRecv error: recv got EOF from 127.0.0.1:48156 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:06:03.388415 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.387646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.1.217.125:41909)
0926 07:06:03.387678 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.387680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.387897 (+ 217us) server_negotiation.cc:407] Connection header received
0926 07:06:03.387988 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.387989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.388000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.388030 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.388370 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48162: BlockingRecv error: recv got EOF from 127.0.0.1:48162 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:06:03.394503 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.393710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48178 (local address 127.1.217.125:41909)
0926 07:06:03.393749 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.393751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.393992 (+ 241us) server_negotiation.cc:407] Connection header received
0926 07:06:03.394089 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.394090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.394101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.394129 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.394452 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48178: BlockingRecv error: recv got EOF from 127.0.0.1:48178 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 07:06:03.404582 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.403721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.1.217.125:41909)
0926 07:06:03.403758 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.403760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.404019 (+ 259us) server_negotiation.cc:407] Connection header received
0926 07:06:03.404112 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.404112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.404123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.404151 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.404524 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48190: BlockingRecv error: recv got EOF from 127.0.0.1:48190 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:06:03.422852 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.421901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48200 (local address 127.1.217.125:41909)
0926 07:06:03.421944 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.421946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.422230 (+ 284us) server_negotiation.cc:407] Connection header received
0926 07:06:03.422359 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.422361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.422373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.422404 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.422794 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48200: BlockingRecv error: recv got EOF from 127.0.0.1:48200 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:06:03.458278 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.456208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.1.217.125:41909)
0926 07:06:03.456254 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.456256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.457623 (+ 1367us) server_negotiation.cc:407] Connection header received
0926 07:06:03.457793 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.457794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.457807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.457837 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.458218 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48206: BlockingRecv error: recv got EOF from 127.0.0.1:48206 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:06:03.524338 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.523640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48208 (local address 127.1.217.125:41909)
0926 07:06:03.523701 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.523703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.523807 (+ 104us) server_negotiation.cc:407] Connection header received
0926 07:06:03.523854 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.523855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.523866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.523930 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.524229 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48208: BlockingRecv error: recv got EOF from 127.0.0.1:48208 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 07:06:03.654449 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.653578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.1.217.125:41909)
0926 07:06:03.653678 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.653681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.653821 (+ 140us) server_negotiation.cc:407] Connection header received
0926 07:06:03.653914 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.653916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.653929 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.654007 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.654359 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48216: BlockingRecv error: recv got EOF from 127.0.0.1:48216 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 07:06:03.912289 24218 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:03.911546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.1.217.125:41909)
0926 07:06:03.911615 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:03.911618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:03.911653 (+ 35us) server_negotiation.cc:407] Connection header received
0926 07:06:03.911728 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:03.911729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:03.911741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:03.911812 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:03.912223 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48224: BlockingRecv error: recv got EOF from 127.0.0.1:48224 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 07:06:04.272713 24321 catalog_manager.cc:1259] Loaded cluster ID: 908501e54efa49cb8264991193a47104
I20260926 07:06:04.272871 24321 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: loading cluster ID for follower catalog manager: success
I20260926 07:06:04.273769 24321 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: acquiring CA information for follower catalog manager: success
I20260926 07:06:04.274103 24321 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91907a1f5f6e4a1caea3563975f8cb66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:06:04.318079 24350 catalog_manager.cc:1259] Loaded cluster ID: 908501e54efa49cb8264991193a47104
I20260926 07:06:04.318154 24350 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: loading cluster ID for follower catalog manager: success
I20260926 07:06:04.318907 24350 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: acquiring CA information for follower catalog manager: success
I20260926 07:06:04.319164 24350 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03b9daf87f8747dd8a1616354ecd11a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:06:04.428179 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:04.425503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.1.217.125:41909)
0926 07:06:04.425614 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:04.425616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:04.425631 (+ 15us) server_negotiation.cc:407] Connection header received
0926 07:06:04.425740 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:04.425741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:04.425802 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:04.425891 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:04.426037 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.426546 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.427327 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.427529 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.427604 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:04.427785 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:04.427908 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:04.427993 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:04.428040 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 07:06:04.428064 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260926 07:06:04.432363 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:04.430037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41628 (local address 127.1.217.124:41325)
0926 07:06:04.430151 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:04.430153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:04.430165 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:06:04.430259 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:04.430260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:04.430293 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:04.430372 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:04.430458 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.430936 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.431592 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.431872 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.431935 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:04.432053 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:04.432149 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:04.432204 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:04.432238 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 07:06:04.432253 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":46,"threads_started":1}
I20260926 07:06:04.436801 24430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:04.434154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.1.217.124:41325)
0926 07:06:04.434198 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:04.434200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:04.434206 (+ 6us) server_negotiation.cc:407] Connection header received
0926 07:06:04.434247 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:04.434248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:04.434257 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:04.434312 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:04.434465 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.435053 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.435983 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.436269 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.436306 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:04.436513 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:04.436548 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:04.436607 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:04.436669 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 07:06:04.436686 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:06:04.437215 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:04.434207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.1.217.125:41909)
0926 07:06:04.434266 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:04.434268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:04.434427 (+ 159us) server_negotiation.cc:407] Connection header received
0926 07:06:04.434508 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:04.434509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:04.434519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:04.434571 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:04.434663 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.435404 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.436394 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.436679 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.436745 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:04.436931 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:04.436961 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:04.436988 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:04.437116 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 07:06:04.437137 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 07:06:04.437312 24437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:04.434015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.1.217.126:42549)
0926 07:06:04.434128 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:04.434130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:04.434363 (+ 233us) server_negotiation.cc:407] Connection header received
0926 07:06:04.434448 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:04.434449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:04.434499 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:04.434591 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:04.434792 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.435513 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.436271 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:04.436565 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:04.436633 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:04.436887 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:04.436987 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:04.437043 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:04.437200 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 07:06:04.437222 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260926 07:06:04.439004 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24071
I20260926 07:06:04.444679 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24146
I20260926 07:06:04.449404 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24221
2026-09-26T07:06:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2837 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 07:06:04 dist-test-slave-t1mc krb5kdc[24441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 07:06:04 dist-test-slave-t1mc krb5kdc[24441](info): set up 2 sockets
Sep 26 07:06:04 dist-test-slave-t1mc krb5kdc[24441](info): commencing operation
krb5kdc: starting...
W20260926 07:06:05.477059 1893 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/test-user.keytab.
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T07:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T07:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.1.217.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
Entry for principal kudu/127.1.217.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab.
I20260926 07:06:05.593683 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:44765
--webserver_interface=127.1.217.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 07:06:05.673501 24454 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:05.673741 24454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:05.673771 24454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:05.673812 24454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:05.674921 24454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:05.674986 24454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:05.675010 24454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:05.675032 24454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:05.675060 24454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:05.676529 24454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.126.keytab
--principal=kudu/127.1.217.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.126:44765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.1.217.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:05.676815 24454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:05.677047 24454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:05.681945 24460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.682086 24462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.681972 24459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:05.682168 24454 server_base.cc:1034] running on GCE node
I20260926 07:06:05.682397 24454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:05.682648 24454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:05.683805 24454 hybrid_clock.cc:648] HybridClock initialized: now 1790406365683798 us; error 33 us; skew 500 ppm
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:05.686488 24454 init.cc:377] Logged in from keytab as kudu/127.1.217.126@KRBTEST.COM (short username kudu)
I20260926 07:06:05.686950 24454 webserver.cc:460] Webserver started at http://127.1.217.126:42395/ using document root <none> and password file <none>
I20260926 07:06:05.687201 24454 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:05.687273 24454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:05.687420 24454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:05.688588 24454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data/instance:
uuid: "567a4101aa8044f595e471e8d532567c"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "c075ffefe15052b3cd290546a168bb9c"
server_key_iv: "ceb4710d1315cca8d6fa78a8684b0ab6"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.688992 24454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance:
uuid: "567a4101aa8044f595e471e8d532567c"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "c075ffefe15052b3cd290546a168bb9c"
server_key_iv: "ceb4710d1315cca8d6fa78a8684b0ab6"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.690433 24454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 07:06:05.691184 24469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:05.691509 24454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:06:05.691622 24454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal
uuid: "567a4101aa8044f595e471e8d532567c"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "c075ffefe15052b3cd290546a168bb9c"
server_key_iv: "ceb4710d1315cca8d6fa78a8684b0ab6"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.691699 24454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:05.703822 24454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:05.705757 24454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:05.705921 24454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:05.709852 24454 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.126:44765
I20260926 07:06:05.710011 24521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.126:44765 every 8 connection(s)
I20260926 07:06:05.710275 24454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/data/info.pb
I20260926 07:06:05.711225 24522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:05.712884 24522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:05.718312 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24454
I20260926 07:06:05.718400 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-0/wal/instance
I20260926 07:06:05.718632 1893 external_mini_cluster.cc:1428] Setting key ea5fd5c5cb7a7899e7032f6c8b4291b6
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:05.722671 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.713572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44765 (local address 127.0.0.1:59802)
0926 07:06:05.713738 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.713776 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:05.714227 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:05.714540 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:05.714543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:05.714777 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:05.714926 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.714930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.715898 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.715899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.716567 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.716568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.717103 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.717104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.717160 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.717712 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:05.717719 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:05.719168 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:05.721221 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.721227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.721228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.721409 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.721680 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.721680 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.721681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.721755 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.722196 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:05.722201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:05.722256 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 07:06:05.722323 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 07:06:05.722442 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260926 07:06:05.722676 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.713592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.1.217.126:44765)
0926 07:06:05.713751 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:05.713767 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:05.713903 (+ 136us) server_negotiation.cc:407] Connection header received
0926 07:06:05.714382 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:05.714384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:05.714401 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:05.714511 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:05.715017 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.715817 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.716648 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.717017 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.717097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.719247 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:05.719254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:05.719255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:05.719262 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:05.721134 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.721510 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.721512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.721513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.721613 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.721818 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.721819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.721820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.722120 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:05.722204 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:05.722387 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 07:06:05.722522 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260926 07:06:05.723656 24522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:05.723994 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.723895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56506)
0926 07:06:05.723928 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.723975 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 07:06:05.724159 24471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
I20260926 07:06:05.724557 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.724472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56510)
0926 07:06:05.724518 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.724542 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 07:06:05.724678 24522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34675: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.1.217.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
Entry for principal kudu/127.1.217.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab.
I20260926 07:06:05.742900 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34675
--webserver_interface=127.1.217.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:06:05.773932 24522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } attempt: 1
I20260926 07:06:05.774469 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.774336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56524)
0926 07:06:05.774399 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.774440 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 07:06:05.774853 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.774770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56530)
0926 07:06:05.774813 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.774837 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 07:06:05.774969 24522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34675: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
W20260926 07:06:05.823720 24529 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:05.823948 24529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:05.823966 24529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:05.823988 24529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 07:06:05.825080 24529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:05.825115 24529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:05.825129 24529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:05.825142 24529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:05.825155 24529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:05.826581 24529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.125.keytab
--principal=kudu/127.1.217.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.125:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.1.217.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:05.826841 24529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:05.827020 24529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:05.831781 24534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.831779 24537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.831779 24535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:05.831884 24529 server_base.cc:1034] running on GCE node
I20260926 07:06:05.832247 24529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:05.832551 24529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:05.833714 24529 hybrid_clock.cc:648] HybridClock initialized: now 1790406365833695 us; error 33 us; skew 500 ppm
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:05.836143 24529 init.cc:377] Logged in from keytab as kudu/127.1.217.125@KRBTEST.COM (short username kudu)
I20260926 07:06:05.836624 24529 webserver.cc:460] Webserver started at http://127.1.217.125:44665/ using document root <none> and password file <none>
I20260926 07:06:05.836936 24529 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:05.837005 24529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:05.837142 24529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:05.838191 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data/instance:
uuid: "c05f74cf6d0d4cfe864c144f97be98f9"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "785ebfe349e4d4f75a45288f2d402bb2"
server_key_iv: "5b3fb196007026b661ee462f2ab1fc1d"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.838553 24529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal/instance:
uuid: "c05f74cf6d0d4cfe864c144f97be98f9"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "785ebfe349e4d4f75a45288f2d402bb2"
server_key_iv: "5b3fb196007026b661ee462f2ab1fc1d"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.839835 24529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 07:06:05.840546 24544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:05.840728 24529 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 07:06:05.840848 24529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal
uuid: "c05f74cf6d0d4cfe864c144f97be98f9"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "785ebfe349e4d4f75a45288f2d402bb2"
server_key_iv: "5b3fb196007026b661ee462f2ab1fc1d"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.840937 24529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:05.843132 24522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } attempt: 2
I20260926 07:06:05.843534 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.843410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56540)
0926 07:06:05.843466 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.843496 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 07:06:05.843909 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.843825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56550)
0926 07:06:05.843869 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.843893 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 07:06:05.844019 24522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.125:34675: Network error: Client connection negotiation failed: client connection to 127.1.217.125:34675: connect: Connection refused (error 111)
I20260926 07:06:05.860299 24529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:05.862282 24529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:05.862442 24529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:05.866602 24529 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.125:34675
I20260926 07:06:05.866639 24596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.125:34675 every 8 connection(s)
I20260926 07:06:05.866974 24529 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/data/info.pb
I20260926 07:06:05.867323 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24529
I20260926 07:06:05.867437 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-1/wal/instance
I20260926 07:06:05.867633 1893 external_mini_cluster.cc:1428] Setting key 527495c963cefedd706f02a5076a0198
I20260926 07:06:05.868157 24597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:05.870270 24597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.1.217.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.1.217.124@KRBTEST.COM" created.
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:05.880918 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.870997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59810 (local address 127.1.217.126:44765)
0926 07:06:05.871056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:05.871058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:05.871245 (+ 187us) server_negotiation.cc:407] Connection header received
0926 07:06:05.871783 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:05.871784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:05.871801 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:05.871864 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:05.872469 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.873019 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.874024 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.874225 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.874311 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.877940 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:05.877941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:05.877942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:05.877950 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:05.879488 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.879906 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.879907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.879907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.879970 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.880230 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.880231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.880231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.880353 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:05.880430 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:05.880708 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 07:06:05.880824 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 07:06:05.881289 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.870975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44765 (local address 127.0.0.1:59810)
0926 07:06:05.871131 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.871165 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:05.871676 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:05.871934 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:05.871938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:05.872188 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:05.872376 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.872380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.873111 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.873112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.873918 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.873920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.874330 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.874332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.874456 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.874994 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:05.875003 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:05.877849 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:05.879592 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.879599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.879601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.879827 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.880039 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.880040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.880040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.880152 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.880423 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:05.880429 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:05.880570 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 07:06:05.880681 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 07:06:05.880859 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260926 07:06:05.882153 24597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:05.889776 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.882431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56558)
0926 07:06:05.882477 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.882487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:05.882529 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:05.882814 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:05.882815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:05.882932 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:05.882977 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.882978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.884281 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.884282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.884764 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.884819 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.885245 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.885246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.885299 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.885748 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:05.885752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:05.887279 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:05.888817 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.888818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.888819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.889015 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.889248 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.889248 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.889249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.889299 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.889523 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:05.889524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:05.889575 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:06:05.889648 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 07:06:05.889715 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 07:06:05.889925 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.882487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.1.217.125:34675)
0926 07:06:05.882653 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:05.882658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:05.882679 (+ 21us) server_negotiation.cc:407] Connection header received
0926 07:06:05.882705 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:05.882707 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:05.882722 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:05.882807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:05.883055 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.884199 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.884888 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.885180 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.885237 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.887351 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:05.887357 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:05.887357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:05.887363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:05.888704 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.889111 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.889114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.889114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.889177 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.889342 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.889342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.889342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.889465 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:05.889531 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:05.889722 (+ 191us) server_negotiation.cc:299] Negotiation successful
0926 07:06:05.889782 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260926 07:06:05.890482 24597 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:05.890784 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.890696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:57978)
0926 07:06:05.890725 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.890765 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 07:06:05.890925 24547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
Entry for principal kudu/127.1.217.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab.
I20260926 07:06:05.891348 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.891257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:57994)
0926 07:06:05.891296 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.891318 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 07:06:05.891470 24597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41917: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
I20260926 07:06:05.891846 1893 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
/tmp/dist-test-tasktn8ZA6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:41917
--webserver_interface=127.1.217.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.1.217.84:46577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 07:06:05.935181 24522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } attempt: 3
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:05.940706 24597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } attempt: 1
I20260926 07:06:05.941108 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.940993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58000)
0926 07:06:05.941048 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.941082 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 07:06:05.941880 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.941771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58004)
0926 07:06:05.941823 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.941849 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:06:05.942203 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.935569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56562)
0926 07:06:05.935643 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.935672 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:05.935734 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:05.935891 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:05.935892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:05.936061 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:05.936125 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.936127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.936855 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.936857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.937396 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:05.937398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.937803 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.937804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:05.937880 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.938342 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:05.938347 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:05.939507 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:05.941184 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.941185 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.941186 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.941374 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.941590 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:05.941591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:05.941591 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:05.941657 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:05.941924 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:05.941925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:05.941977 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 07:06:05.942063 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 07:06:05.942135 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 07:06:05.942248 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.935636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.1.217.125:34675)
0926 07:06:05.935713 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:05.935716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:05.935728 (+ 12us) server_negotiation.cc:407] Connection header received
0926 07:06:05.935804 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:05.935805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:05.935824 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:05.935886 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:05.936232 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.936732 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.937495 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:05.937720 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:05.937794 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:05.939594 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:05.939595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:05.939596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:05.939603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:05.941091 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.941446 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.941448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.941448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.941515 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:05.941722 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:05.941723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:05.941723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:05.941853 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:05.941919 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:05.942089 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 07:06:05.942164 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 07:06:05.942366 24597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41917: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
I20260926 07:06:05.942519 24522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:05.942816 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.942707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58020)
0926 07:06:05.942759 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.942785 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 07:06:05.943141 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.943057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58030)
0926 07:06:05.943098 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.943121 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 07:06:05.943235 24522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41917: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
W20260926 07:06:05.973097 24604 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 07:06:05.973331 24604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 07:06:05.973361 24604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 07:06:05.973416 24604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 07:06:05.974366 24522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } attempt: 1
W20260926 07:06:05.974604 24604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 07:06:05.974661 24604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 07:06:05.974686 24604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 07:06:05.974713 24604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 07:06:05.974732 24604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 07:06:05.974831 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.974668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58034)
0926 07:06:05.974741 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.974797 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 07:06:05.975195 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:05.975090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58036)
0926 07:06:05.975140 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:05.975170 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 07:06:05.975302 24522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41917: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
I20260926 07:06:05.976862 24604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.217.84:46577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.1.217.126:44765,127.1.217.125:34675,127.1.217.124:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/krb5kdc/kudu_127.1.217.124.keytab
--principal=kudu/127.1.217.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.217.124:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.1.217.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 06:04:27 UTC on ccycloud.quasar-krtrcq.root.comops.site
build id 4673739
I20260926 07:06:05.977128 24604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 07:06:05.977383 24604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 07:06:05.982525 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.982582 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 07:06:05.982738 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 07:06:05.982779 24604 server_base.cc:1034] running on GCE node
I20260926 07:06:05.983074 24604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 07:06:05.983398 24604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 07:06:05.984553 24604 hybrid_clock.cc:648] HybridClock initialized: now 1790406365984534 us; error 35 us; skew 500 ppm
Sep 26 07:06:05 dist-test-slave-t1mc krb5kdc[24441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 07:06:05.987145 24604 init.cc:377] Logged in from keytab as kudu/127.1.217.124@KRBTEST.COM (short username kudu)
I20260926 07:06:05.987510 24604 webserver.cc:460] Webserver started at http://127.1.217.124:38729/ using document root <none> and password file <none>
I20260926 07:06:05.987684 24604 fs_manager.cc:359] Metadata directory not provided
I20260926 07:06:05.987722 24604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 07:06:05.987808 24604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 07:06:05.988823 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data/instance:
uuid: "cb97cf93f60e47789162f8c3a0124261"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "b64754bad25fec5f5d71b1e39d0f093b"
server_key_iv: "5cd6b53e96e6db7551efb9e7d5e40b05"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.989151 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal/instance:
uuid: "cb97cf93f60e47789162f8c3a0124261"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "b64754bad25fec5f5d71b1e39d0f093b"
server_key_iv: "5cd6b53e96e6db7551efb9e7d5e40b05"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.990473 24604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:06:05.991248 24619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 07:06:05.991425 24604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 07:06:05.991544 24604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data,/tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal
uuid: "cb97cf93f60e47789162f8c3a0124261"
format_stamp: "Formatted at 2026-09-26 07:06:05 on dist-test-slave-t1mc"
server_key: "b64754bad25fec5f5d71b1e39d0f093b"
server_key_iv: "5cd6b53e96e6db7551efb9e7d5e40b05"
server_key_version: "encryptionkey@0"
I20260926 07:06:05.991596 24604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 07:06:06.010525 24597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } attempt: 2
I20260926 07:06:06.011010 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.010875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58038)
0926 07:06:06.010940 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.010982 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 07:06:06.011410 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.011304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58054)
0926 07:06:06.011351 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.011379 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 07:06:06.011560 24597 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.1.217.124:41917: Network error: Client connection negotiation failed: client connection to 127.1.217.124:41917: connect: Connection refused (error 111)
I20260926 07:06:06.017676 24604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 07:06:06.019598 24604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 07:06:06.019757 24604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 07:06:06.023553 24604 rpc_server.cc:307] RPC server started. Bound to: 127.1.217.124:41917
I20260926 07:06:06.023607 24671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.1.217.124:41917 every 8 connection(s)
I20260926 07:06:06.023938 24604 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/data/info.pb
I20260926 07:06:06.024966 24672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 07:06:06.026631 1893 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu as pid 24604
I20260926 07:06:06.026764 1893 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktn8ZA6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790406101908495-1893-0/minicluster-data/master-2/wal/instance
I20260926 07:06:06.026615 24672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:06.026953 1893 external_mini_cluster.cc:1428] Setting key 9c6d7e90f875c675775b9bc9b7252311
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:06.036700 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.027400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.1.217.126:44765)
0926 07:06:06.027447 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.027450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.027728 (+ 278us) server_negotiation.cc:407] Connection header received
0926 07:06:06.028240 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.028241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.028252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.028313 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.028911 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.029378 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.030291 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.030489 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.030564 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.032945 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.032947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.032947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.032956 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.035067 (+ 2111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.035629 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.035630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.035631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.035710 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.035982 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.035983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.035984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.036129 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.036190 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.036484 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.036584 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 07:06:06.037177 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.027388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58060 (local address 127.1.217.124:41917)
0926 07:06:06.027612 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.027621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.027652 (+ 31us) server_negotiation.cc:407] Connection header received
0926 07:06:06.028125 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.028135 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.028168 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.028261 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.028962 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.029731 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.030455 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.030768 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.030853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.033555 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.033561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.033561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.033567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.035440 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.035828 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.035831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.035831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.035908 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.036107 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.036108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.036108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.036370 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.036514 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.036622 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.036702 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":194,"thread_start_us":131,"threads_started":1}
I20260926 07:06:06.037177 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.027381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.126:44765 (local address 127.0.0.1:59822)
0926 07:06:06.027586 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.027626 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:06.028130 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:06.028343 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:06.028347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:06.028613 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:06.028811 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.028817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.029479 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.029480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.030189 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.030191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.030592 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.030594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.030708 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.031204 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:06.031214 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:06.032860 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:06.035149 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.035154 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.035157 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.035553 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.035780 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.035780 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.035780 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.035911 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.036199 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:06.036204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:06.036360 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 07:06:06.036452 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 07:06:06.036657 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":19,"thread_start_us":111,"threads_started":1}
I20260926 07:06:06.037732 24672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.126@KRBTEST.COM
I20260926 07:06:06.045279 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.038065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.1.217.125:34675)
0926 07:06:06.038108 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.038111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.038148 (+ 37us) server_negotiation.cc:407] Connection header received
0926 07:06:06.038223 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.038224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.038236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.038293 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.038602 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.039212 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.039944 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.040270 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.040329 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.042138 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.042139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.042140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.042146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.044125 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044461 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.044462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.044462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.044518 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044723 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.044724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.044724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.044888 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.044961 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.045112 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.045181 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:06:06.045280 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.037995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.125:34675 (local address 127.0.0.1:56566)
0926 07:06:06.038106 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.038117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:06.038163 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:06.038321 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:06.038322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:06.038479 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:06.038526 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.038528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.039286 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.039287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.039869 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.039871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.040342 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.040343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.040410 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.040866 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:06.040870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:06.042042 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:06.044210 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.044212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.044379 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.044583 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.044585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.044641 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.044953 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:06.044954 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:06.044998 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:06:06.045087 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 07:06:06.045181 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 07:06:06.045775 24672 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } has no permanent_uuid. Determining permanent_uuid...
I20260926 07:06:06.045938 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.038063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.1.217.126:44765)
0926 07:06:06.038113 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.038115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.038302 (+ 187us) server_negotiation.cc:407] Connection header received
0926 07:06:06.038346 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.038347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.038357 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.038411 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.038939 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.039491 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.040234 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.040477 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.040551 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.042724 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.042725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.042725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.042732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.044294 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044649 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.044651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.044651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.044735 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.044995 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.044996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.044996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.045106 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.045292 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.045762 (+ 470us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.045843 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.124@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:06.050421 24522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } attempt: 2
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.1.217.125@KRBTEST.COM
I20260926 07:06:06.053823 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.046128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.1.217.124:41917)
0926 07:06:06.046165 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.046167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.046178 (+ 11us) server_negotiation.cc:407] Connection header received
0926 07:06:06.046201 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.046202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.046211 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.046257 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.046551 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.047285 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.048113 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.048414 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.048481 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.050177 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.050178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.050179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.050186 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.051679 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.053069 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.053070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.053070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.053127 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.053295 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.053295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.053295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.053405 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.053441 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.053621 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.053724 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 07:06:06.054104 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.046052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58070)
0926 07:06:06.046117 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.046126 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:06.046159 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:06.046291 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:06.046292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:06.046440 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:06.046472 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.046474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.047354 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.047355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.048041 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.048043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.048483 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.048484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.048521 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.048958 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:06.048963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:06.050096 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:06.051773 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.051774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.051774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.052992 (+ 1218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.053187 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.053188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.053188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.053238 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.053479 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:06.053480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:06.053512 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 07:06:06.053931 (+ 419us) client_negotiation.cc:240] Negotiation successful
0926 07:06:06.054019 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 07:06:06.054471 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.046514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56570 (local address 127.1.217.125:34675)
0926 07:06:06.046547 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.046549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.046557 (+ 8us) server_negotiation.cc:407] Connection header received
0926 07:06:06.046602 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.046603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.046610 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.046649 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.047164 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.047816 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.048649 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.048978 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.049059 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.051469 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.051470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.051471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.051477 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.052922 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.053292 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.053293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.053294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.053351 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.053549 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.053549 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.053550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.053654 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.053837 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.054269 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.054350 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.126@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:06.055881 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: Bootstrap starting.
I20260926 07:06:06.056523 24672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:06.056885 24672 log.cc:824] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:06.057700 1893 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 07:06:06.057777 24672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: No bootstrap required, opened a new log
I20260926 07:06:06.058876 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.050781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58076)
0926 07:06:06.050834 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.050843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:06.050879 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:06.051309 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:06.051310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:06.051471 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:06.051517 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.051519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.052483 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.052484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.053066 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.053068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.054001 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.054002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.054061 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.054458 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:06.054463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:06.055801 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:06.057819 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.057819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.057820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.058022 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.058224 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.058225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.058226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.058282 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.058595 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:06.058596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:06.058640 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 07:06:06.058689 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 07:06:06.058783 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 07:06:06.059288 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.050846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.1.217.124:41917)
0926 07:06:06.050993 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.050994 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.051003 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:06:06.051209 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.051210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.051220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.051305 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.051604 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.052396 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.053145 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.053905 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.054224 (+ 319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.055876 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.055877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.055877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.055882 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.057733 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.058093 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.058094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.058094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.058149 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.058350 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.058351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.058351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.058484 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.059050 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.059090 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.059168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 07:06:06.058630 24496 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "567a4101aa8044f595e471e8d532567c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59836
I20260926 07:06:06.059799 24672 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.059952 24672 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb97cf93f60e47789162f8c3a0124261, State: Initialized, Role: FOLLOWER
I20260926 07:06:06.060039 24672 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.060329 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.060400 24672 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:06.060420 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.060595 24522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: Bootstrap starting.
I20260926 07:06:06.061305 24522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:06.061584 24522 log.cc:824] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: Log is configured to *not* fsync() on all Append() calls
W20260926 07:06:06.062587 24690 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:06.062721 24690 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:06:06.063254 24522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: No bootstrap required, opened a new log
I20260926 07:06:06.062579 24604 master_runner.cc:186] Error getting master registration for 127.1.217.126:44765: OK, instance_id { permanent_uuid: "567a4101aa8044f595e471e8d532567c" instance_seqno: 1790406365708638 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:06:06.064425 24604 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:06:06.064618 24522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.064754 24522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567a4101aa8044f595e471e8d532567c, State: Initialized, Role: FOLLOWER
I20260926 07:06:06.064860 24522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.065160 24522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:06.065133 24693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.065223 24693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.065282 24496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 07:06:06.065330 24496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:06.066052 24496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.066283 24496 leader_election.cc:288] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [CANDIDATE]: Term 1 election: Requested vote from peers c05f74cf6d0d4cfe864c144f97be98f9 (127.1.217.125:34675), cb97cf93f60e47789162f8c3a0124261 (127.1.217.124:41917)
I20260926 07:06:06.066509 24646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "567a4101aa8044f595e471e8d532567c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb97cf93f60e47789162f8c3a0124261"
I20260926 07:06:06.066612 24646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:06.067412 24646 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567a4101aa8044f595e471e8d532567c in term 1.
I20260926 07:06:06.068420 24472 leader_election.cc:302] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 567a4101aa8044f595e471e8d532567c, cb97cf93f60e47789162f8c3a0124261; no voters:
I20260926 07:06:06.068619 24693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 07:06:06.068760 24693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [term 1 LEADER]: Becoming Leader. State: Replica: 567a4101aa8044f595e471e8d532567c, State: Running, Role: LEADER
I20260926 07:06:06.068864 24693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.069274 24709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 567a4101aa8044f595e471e8d532567c. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.069468 24709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: This master's current role is: LEADER
W20260926 07:06:06.069974 24713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:06.070031 24713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 07:06:06.070786 24454 master_runner.cc:186] Error getting master registration for 127.1.217.125:34675: OK, instance_id { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" instance_seqno: 1790406365865469 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 07:06:06.073372 24454 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 07:06:06.074368 24715 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 07:06:06.074729 24715 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 07:06:06.075817 24525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.067756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.1.217.126:44765)
0926 07:06:06.067804 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.067806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.068012 (+ 206us) server_negotiation.cc:407] Connection header received
0926 07:06:06.068103 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.068104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.068114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.068168 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.068959 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.069661 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.070679 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.070960 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.071003 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.072613 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.072614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.072615 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.072621 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.074101 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.074584 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.074585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.074585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.074648 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.074925 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.074926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.074926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.075056 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.075144 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.075633 (+ 489us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.075729 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 07:06:06.077389 24646 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [term 1 FOLLOWER]: Refusing update from remote peer 567a4101aa8044f595e471e8d532567c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:06:06.077741 24693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:06:06.079063 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 567a4101aa8044f595e471e8d532567c. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.079147 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.080480 24693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.080597 24693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: This master's current role is: LEADER
I20260926 07:06:06.080756 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.067556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.1.217.125:34675)
0926 07:06:06.067611 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.067613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.070211 (+ 2598us) server_negotiation.cc:407] Connection header received
0926 07:06:06.070311 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.070313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.070323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.070353 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.073862 (+ 3509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.074595 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.075617 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.075928 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.075971 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.077555 (+ 1584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.077557 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.077557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.077564 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.079238 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.079633 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.079634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.079635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.079934 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.080165 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.080166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.080167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.080296 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.080381 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.080582 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.080662 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 07:06:06.080956 24679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.081040 24679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.081451 24716 mvcc.cc:205] Tried to move back new op lower bound from 7333504475449499648 to 7333504475418669056. Current Snapshot: MvccSnapshot[applied={T|T < 7333504475449499648}]
I20260926 07:06:06.081835 24717 mvcc.cc:205] Tried to move back new op lower bound from 7333504475449499648 to 7333504475418669056. Current Snapshot: MvccSnapshot[applied={T|T < 7333504475449499648}]
I20260926 07:06:06.082779 24715 catalog_manager.cc:1347] Generated new cluster ID: 1e3ab9cf5d9544debd44d7468a470339
I20260926 07:06:06.082834 24715 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 07:06:06.085698 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.067878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.1.217.124:41917)
0926 07:06:06.069527 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.069529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.076382 (+ 6853us) server_negotiation.cc:407] Connection header received
0926 07:06:06.076409 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.076410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.076438 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.076494 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.077144 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.077823 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.078961 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.079283 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.079348 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.082052 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.082053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.082054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.082062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.083991 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.084504 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.084505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.084505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.084574 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.084866 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.084866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.084867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.085027 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.085179 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.085406 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.085566 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630}
I20260926 07:06:06.101287 24715 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 07:06:06.102061 24715 catalog_manager.cc:1504] Loading token signing keys...
I20260926 07:06:06.102660 24597 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } attempt: 3
Sep 26 07:06:06 dist-test-slave-t1mc krb5kdc[24441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790406365, etypes {rep=17 tkt=17 ses=17}, kudu/127.1.217.125@KRBTEST.COM for kudu/127.1.217.124@KRBTEST.COM
I20260926 07:06:06.112200 24600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.102899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.217.124:41917 (local address 127.0.0.1:58106)
0926 07:06:06.102954 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 07:06:06.102968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 07:06:06.103020 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 07:06:06.103341 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 07:06:06.103342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 07:06:06.103527 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 07:06:06.103575 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.103576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.105028 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.105029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.105834 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 07:06:06.105836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.106527 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.106528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 07:06:06.106593 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.107070 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 07:06:06.107076 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 07:06:06.108570 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 07:06:06.111008 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.111009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.111009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.111214 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.111488 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 07:06:06.111488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 07:06:06.111488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 07:06:06.111554 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 07:06:06.111912 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 07:06:06.111913 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 07:06:06.111964 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 07:06:06.112005 (+ 41us) client_negotiation.cc:240] Negotiation successful
0926 07:06:06.112093 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 07:06:06.113062 24677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:06.103096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.1.217.124:41917)
0926 07:06:06.103166 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.103168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.103177 (+ 9us) server_negotiation.cc:407] Connection header received
0926 07:06:06.103226 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.103227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.103239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.103646 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.103726 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.104899 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.105935 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:06.106436 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:06.106777 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:06.108672 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 07:06:06.108674 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 07:06:06.108674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 07:06:06.108680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 07:06:06.110904 (+ 2224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.111301 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.111302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.111302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.111404 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 07:06:06.111624 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 07:06:06.111625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 07:06:06.111625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 07:06:06.111824 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 07:06:06.112639 (+ 815us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:06.112713 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 07:06:06.112876 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:06:06.114694 24597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: Bootstrap starting.
I20260926 07:06:06.115386 24597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: Neither blocks nor log segments found. Creating new log.
I20260926 07:06:06.115640 24597 log.cc:824] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: Log is configured to *not* fsync() on all Append() calls
I20260926 07:06:06.117453 24597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: No bootstrap required, opened a new log
I20260926 07:06:06.119230 24597 raft_consensus.cc:348] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.119349 24597 raft_consensus.cc:729] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05f74cf6d0d4cfe864c144f97be98f9, State: Initialized, Role: FOLLOWER
I20260926 07:06:06.119411 24597 consensus_queue.cc:260] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } }
I20260926 07:06:06.119716 24597 sys_catalog.cc:564] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 07:06:06.120954 24570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 07:06:06.121754 24570 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [term 1 FOLLOWER]: Refusing update from remote peer 567a4101aa8044f595e471e8d532567c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 07:06:06.122108 24571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "567a4101aa8044f595e471e8d532567c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c05f74cf6d0d4cfe864c144f97be98f9"
I20260926 07:06:06.122108 24720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 07:06:06.123536 24571 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 567a4101aa8044f595e471e8d532567c in term 1.
I20260926 07:06:06.125530 24736 mvcc.cc:205] Tried to move back new op lower bound from 7333504475545505792 to 7333504475418669056. Current Snapshot: MvccSnapshot[applied={T|T < 7333504475449499648}]
I20260926 07:06:06.125713 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.125850 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.125944 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 567a4101aa8044f595e471e8d532567c. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.126061 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.126138 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.126255 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 07:06:06.126616 24715 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c: Generated new TSK 0
I20260926 07:06:06.126806 24715 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 07:06:06.127097 24709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "567a4101aa8044f595e471e8d532567c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "567a4101aa8044f595e471e8d532567c" member_type: VOTER last_known_addr { host: "127.1.217.126" port: 44765 } } peers { permanent_uuid: "c05f74cf6d0d4cfe864c144f97be98f9" member_type: VOTER last_known_addr { host: "127.1.217.125" port: 34675 } } peers { permanent_uuid: "cb97cf93f60e47789162f8c3a0124261" member_type: VOTER last_known_addr { host: "127.1.217.124" port: 41917 } } }
I20260926 07:06:06.127218 24709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567a4101aa8044f595e471e8d532567c [sys.catalog]: This master's current role is: LEADER
W20260926 07:06:06.127789 24734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 07:06:06.127882 24734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 07:06:06.155594 24525 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.153979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.1.217.126:44765)
0926 07:06:06.154024 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.154026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.154924 (+ 898us) server_negotiation.cc:407] Connection header received
0926 07:06:06.155030 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.155031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.155043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.155074 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.155512 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59856: BlockingRecv error: recv got EOF from 127.0.0.1:59856 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:06:06.156383 24677 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.154342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.1.217.124:41917)
0926 07:06:06.154382 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.154384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.155870 (+ 1486us) server_negotiation.cc:407] Connection header received
0926 07:06:06.155967 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.155968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.155979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.156008 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.156323 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58114: BlockingRecv error: recv got EOF from 127.0.0.1:58114 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 07:06:06.157231 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.154661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.1.217.125:34675)
0926 07:06:06.154706 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.154708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.156608 (+ 1900us) server_negotiation.cc:407] Connection header received
0926 07:06:06.156796 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.156797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.156808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.156840 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.157170 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56582: BlockingRecv error: recv got EOF from 127.0.0.1:56582 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 07:06:06.159433 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.158734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.1.217.125:34675)
0926 07:06:06.158768 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.158770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.158952 (+ 182us) server_negotiation.cc:407] Connection header received
0926 07:06:06.159053 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.159054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.159072 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.159100 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.159385 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56594: BlockingRecv error: recv got EOF from 127.0.0.1:56594 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 07:06:06.162423 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.161654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.1.217.125:34675)
0926 07:06:06.161684 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.161686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.161966 (+ 280us) server_negotiation.cc:407] Connection header received
0926 07:06:06.162060 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.162061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.162070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.162097 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.162383 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 07:06:06.166258 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.165470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56600 (local address 127.1.217.125:34675)
0926 07:06:06.165503 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.165505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.165745 (+ 240us) server_negotiation.cc:407] Connection header received
0926 07:06:06.165874 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.165875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.165886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.165914 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.166213 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56600: BlockingRecv error: recv got EOF from 127.0.0.1:56600 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:06:06.172012 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.171304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.1.217.125:34675)
0926 07:06:06.171336 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.171338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.171543 (+ 205us) server_negotiation.cc:407] Connection header received
0926 07:06:06.171642 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.171643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.171653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.171681 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.171969 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56608: BlockingRecv error: recv got EOF from 127.0.0.1:56608 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 07:06:06.181951 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.181156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.1.217.125:34675)
0926 07:06:06.181192 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.181194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.181427 (+ 233us) server_negotiation.cc:407] Connection header received
0926 07:06:06.181523 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.181524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.181534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.181562 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.181892 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56614: BlockingRecv error: recv got EOF from 127.0.0.1:56614 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 07:06:06.209647 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.205127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.1.217.125:34675)
0926 07:06:06.205174 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.205176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.208977 (+ 3801us) server_negotiation.cc:407] Connection header received
0926 07:06:06.209093 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.209094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.209109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.209146 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.209558 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56620: BlockingRecv error: recv got EOF from 127.0.0.1:56620 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 07:06:06.243713 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.242895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.1.217.125:34675)
0926 07:06:06.242943 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.242946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.243089 (+ 143us) server_negotiation.cc:407] Connection header received
0926 07:06:06.243175 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.243176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.243195 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.243257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.243645 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56622: BlockingRecv error: recv got EOF from 127.0.0.1:56622 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 07:06:06.309578 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.308861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.1.217.125:34675)
0926 07:06:06.308946 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.308949 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.309043 (+ 94us) server_negotiation.cc:407] Connection header received
0926 07:06:06.309122 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.309123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.309135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.309188 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.309516 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56632: BlockingRecv error: recv got EOF from 127.0.0.1:56632 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 07:06:06.439623 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.438851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56636 (local address 127.1.217.125:34675)
0926 07:06:06.438961 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.438963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.439079 (+ 116us) server_negotiation.cc:407] Connection header received
0926 07:06:06.439135 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.439136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.439146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.439188 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.439561 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56636: BlockingRecv error: recv got EOF from 127.0.0.1:56636 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 07:06:06.697600 24602 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 07:06:06.696980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.1.217.125:34675)
0926 07:06:06.697057 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:06.697059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:06.697066 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:06:06.697090 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:06.697090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:06.697100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:06.697167 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 07:06:06.697524 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56646: BlockingRecv error: recv got EOF from 127.0.0.1:56646 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 07:06:07.063187 24690 catalog_manager.cc:1259] Loaded cluster ID: 1e3ab9cf5d9544debd44d7468a470339
I20260926 07:06:07.063265 24690 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: loading cluster ID for follower catalog manager: success
I20260926 07:06:07.064033 24690 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: acquiring CA information for follower catalog manager: success
I20260926 07:06:07.064282 24690 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb97cf93f60e47789162f8c3a0124261: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:06:07.128362 24734 catalog_manager.cc:1259] Loaded cluster ID: 1e3ab9cf5d9544debd44d7468a470339
I20260926 07:06:07.128473 24734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: loading cluster ID for follower catalog manager: success
I20260926 07:06:07.129405 24734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: acquiring CA information for follower catalog manager: success
I20260926 07:06:07.129712 24734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c05f74cf6d0d4cfe864c144f97be98f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 07:06:07.213397 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:07.210816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.1.217.125:34675)
0926 07:06:07.210930 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:07.210932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:07.210996 (+ 64us) server_negotiation.cc:407] Connection header received
0926 07:06:07.211069 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:07.211070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:07.211121 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:07.211194 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:07.211326 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.211809 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.212549 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.212732 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.212842 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:07.213001 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:07.213120 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:07.213181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:07.213271 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 07:06:07.213287 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260926 07:06:07.217448 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:07.215201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.1.217.124:41917)
0926 07:06:07.215313 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:07.215315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:07.215322 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:06:07.215356 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:07.215356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:07.215388 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:07.215456 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:07.215539 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.216001 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.216695 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.217033 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.217084 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:07.217204 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:07.217301 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:07.217356 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:07.217370 (+ 14us) server_negotiation.cc:299] Negotiation successful
0926 07:06:07.217386 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":66,"threads_started":1}
I20260926 07:06:07.222254 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:07.219389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.1.217.124:41917)
0926 07:06:07.219422 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:07.219424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:07.219431 (+ 7us) server_negotiation.cc:407] Connection header received
0926 07:06:07.219512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:07.219513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:07.219522 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:07.219591 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:07.219697 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.220447 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221451 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.221731 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221799 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:07.221973 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:07.222011 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:07.222071 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:07.222135 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 07:06:07.222156 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 07:06:07.222303 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:07.219151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.1.217.125:34675)
0926 07:06:07.219194 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:07.219196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:07.219368 (+ 172us) server_negotiation.cc:407] Connection header received
0926 07:06:07.219483 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:07.219484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:07.219497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:07.219545 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:07.219724 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.220280 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221481 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.221782 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221841 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:07.222027 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:07.222061 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:07.222090 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:07.222196 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 07:06:07.222219 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 07:06:07.222697 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 07:06:07.219146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.1.217.126:44765)
0926 07:06:07.219250 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 07:06:07.219252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 07:06:07.219374 (+ 122us) server_negotiation.cc:407] Connection header received
0926 07:06:07.219426 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 07:06:07.219428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 07:06:07.219501 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 07:06:07.219585 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 07:06:07.219767 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.220526 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221563 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 07:06:07.221832 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 07:06:07.221899 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 07:06:07.222060 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 07:06:07.222169 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 07:06:07.222449 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0926 07:06:07.222619 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 07:06:07.222640 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
I20260926 07:06:07.228600 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24454
I20260926 07:06:07.233682 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24529
I20260926 07:06:07.238600 1893 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktn8ZA6/build/release/bin/kudu with pid 24604
2026-09-26T07:06:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2789 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5627 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (265337 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST